[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:41 dist-test-slave-286n krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:41 dist-test-slave-286n krb5kdc[399](info): set up 2 sockets
Sep 28 08:58:41 dist-test-slave-286n krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 08:58:42.738655 392 mini_kdc.cc:121] Time spent starting KDC: real 2.580s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 08:58:42.877614 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.139s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 08:58:43.071414 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.194s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/test-user.keytab.
W20260928 08:58:43.881275 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.724s user 0.000s sys 0.001s
Sep 28 08:58:43 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260928 08:58:44.308140 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.405s user 0.000s sys 0.002s
I20260928 08:58:44.308575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:44.390015 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:44.390225 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:44.390247 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:44.391304 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:44.391336 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:44.391350 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:44.391363 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:44.391374 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:44.392866 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:44.393057 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:44.393246 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:44.398458 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:44.398567 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:44.398458 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.398586 412 server_base.cc:1034] running on GCE node
I20260928 08:58:44.398931 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:44.399242 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:44.400400 412 hybrid_clock.cc:648] HybridClock initialized: now 1790585924400383 us; error 32 us; skew 500 ppm
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:44.409947 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:44.410394 412 webserver.cc:460] Webserver started at http://127.0.98.62:41981/ using document root <none> and password file <none>
I20260928 08:58:44.410674 412 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:44.410718 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:44.410807 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:44.411811 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "fb63de36fd7c4d13883370c3235cb08b"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "5e1f7d04a4c0b6599aa1c06e6491908c"
server_key_iv: "1e4fa4e462ee43954df080edd80154d8"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.412122 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb63de36fd7c4d13883370c3235cb08b"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "5e1f7d04a4c0b6599aa1c06e6491908c"
server_key_iv: "1e4fa4e462ee43954df080edd80154d8"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.413522 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:58:44.414233 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.414404 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:44.414508 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "fb63de36fd7c4d13883370c3235cb08b"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "5e1f7d04a4c0b6599aa1c06e6491908c"
server_key_iv: "1e4fa4e462ee43954df080edd80154d8"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.414580 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:44.424692 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:44.426424 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:44.426584 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:44.430442 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35787
I20260928 08:58:44.430472 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35787 every 8 connection(s)
I20260928 08:58:44.430876 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:44.431725 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:44.433385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 412
I20260928 08:58:44.433506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:44.434289 392 external_mini_cluster.cc:1428] Setting key 7435572e8eea9c73b08bea444ebbbaa6
I20260928 08:58:44.434381 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b: Bootstrap starting.
I20260928 08:58:44.435065 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:44.435456 480 log.cc:824] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:44.436245 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b: No bootstrap required, opened a new log
I20260928 08:58:44.438405 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb63de36fd7c4d13883370c3235cb08b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260928 08:58:44.438575 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:44.438612 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb63de36fd7c4d13883370c3235cb08b, State: Initialized, Role: FOLLOWER
I20260928 08:58:44.438772 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb63de36fd7c4d13883370c3235cb08b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260928 08:58:44.438866 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:44.438900 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:44.438948 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:44.439997 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb63de36fd7c4d13883370c3235cb08b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260928 08:58:44.440115 480 leader_election.cc:302] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb63de36fd7c4d13883370c3235cb08b; no voters:
I20260928 08:58:44.440291 480 leader_election.cc:288] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:44.440363 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:44.440560 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [term 1 LEADER]: Becoming Leader. State: Replica: fb63de36fd7c4d13883370c3235cb08b, State: Running, Role: LEADER
I20260928 08:58:44.440587 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:44.440721 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb63de36fd7c4d13883370c3235cb08b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } }
I20260928 08:58:44.442207 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb63de36fd7c4d13883370c3235cb08b. Latest consensus state: current_term: 1 leader_uuid: "fb63de36fd7c4d13883370c3235cb08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb63de36fd7c4d13883370c3235cb08b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } } }
I20260928 08:58:44.442309 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:44.442387 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb63de36fd7c4d13883370c3235cb08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb63de36fd7c4d13883370c3235cb08b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35787 } } }
I20260928 08:58:44.442440 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:44.442634 499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:44.443379 499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:44.444999 499 catalog_manager.cc:1347] Generated new cluster ID: 0fce7680da534c0f9d2050b43b393281
I20260928 08:58:44.445051 499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:44.451599 499 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:44.452265 499 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:58:44.453344 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.435228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.0.98.62:35787)
0928 08:58:44.435430 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:44.435439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:44.435666 (+ 227us) server_negotiation.cc:407] Connection header received
0928 08:58:44.438388 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:44.438394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:44.438440 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:44.438552 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:44.440824 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.441924 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.443511 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.444079 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.444219 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.448186 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:44.448190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:44.448191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:44.448201 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:44.450823 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.451333 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.451336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.451337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.451486 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.451801 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.451802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.451802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.452045 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:44.452180 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:44.452977 (+ 797us) server_negotiation.cc:299] Negotiation successful
0928 08:58:44.453116 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260928 08:58:44.457185 499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb63de36fd7c4d13883370c3235cb08b: Generated new TSK 0
I20260928 08:58:44.457381 499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:58:44.484112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35787
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:44.566416 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:44.566655 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:44.566696 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:44.568022 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:44.568105 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:44.568152 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:58:44.570075 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:44.570350 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:44.570585 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:44.575536 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.575614 506 server_base.cc:1034] running on GCE node
W20260928 08:58:44.575621 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:44.575518 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.575933 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:44.576186 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:44.577344 506 hybrid_clock.cc:648] HybridClock initialized: now 1790585924577328 us; error 26 us; skew 500 ppm
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:44.579656 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:58:44.580098 506 webserver.cc:460] Webserver started at http://127.0.98.1:45115/ using document root <none> and password file <none>
I20260928 08:58:44.580309 506 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:44.580374 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:44.580516 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:44.581604 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "2d0577bbbc484f94bf2db4460047b563"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "8a0a84028797c73f957b19ac500787dd"
server_key_iv: "0aa8e18b134b11d3a813292987396126"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.581971 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2d0577bbbc484f94bf2db4460047b563"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "8a0a84028797c73f957b19ac500787dd"
server_key_iv: "0aa8e18b134b11d3a813292987396126"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.583280 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:44.583983 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.584146 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:44.584239 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "2d0577bbbc484f94bf2db4460047b563"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "8a0a84028797c73f957b19ac500787dd"
server_key_iv: "0aa8e18b134b11d3a813292987396126"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.584321 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:44.596138 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:44.598022 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:44.598176 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:44.598387 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:44.598673 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:44.598731 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.598773 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:44.598809 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.604781 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34153
I20260928 08:58:44.604848 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34153 every 8 connection(s)
I20260928 08:58:44.605160 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:58:44.608897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 506
I20260928 08:58:44.608994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:58:44.609163 392 external_mini_cluster.cc:1428] Setting key a020ae28adbded15bf5133867a2dadf7
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:44.614548 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.605804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35789 (local address 127.0.98.62:35787)
0928 08:58:44.605883 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:44.605885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:44.606041 (+ 156us) server_negotiation.cc:407] Connection header received
0928 08:58:44.606571 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:44.606572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:44.606586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:44.606646 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:44.607555 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.608077 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.609199 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.609476 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.609553 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.611679 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:44.611680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:44.611681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:44.611688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:44.613153 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.613639 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.613640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.613640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.613692 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.613958 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.613958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.613959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.614053 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:44.614102 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:44.614421 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 08:58:44.614488 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:58:44.614877 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.605768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35787 (local address 127.0.98.1:35789)
0928 08:58:44.605925 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:44.605963 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:44.606469 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:44.606715 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:44.606719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:44.607278 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:44.607461 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:44.607466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.608172 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.608173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:44.609112 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:44.609115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.609562 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.609563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:44.609683 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.610238 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:44.610246 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:44.611592 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:44.613247 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:44.613254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:44.613256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:44.613566 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:44.613756 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:44.613757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:44.613757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:44.613892 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:44.614112 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:44.614118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:44.614286 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 08:58:44.614410 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 08:58:44.614590 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260928 08:58:44.615402 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35787
I20260928 08:58:44.615506 635 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:44.615686 635 heartbeater.cc:511] Master 127.0.98.62:35787 requested a full tablet report, sending...
I20260928 08:58:44.616168 444 ts_manager.cc:194] Registered new tserver with Master: 2d0577bbbc484f94bf2db4460047b563 (127.0.98.1:34153)
I20260928 08:58:44.616878 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35789
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:58:44.635363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35787
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:44.716199 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:44.716437 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:44.716476 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:44.717734 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:44.717793 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:44.717839 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:58:44.719322 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:44.719579 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:44.719798 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:44.725114 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.725145 640 server_base.cc:1034] running on GCE node
W20260928 08:58:44.725114 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:44.725169 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.725544 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:44.725752 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:44.726884 640 hybrid_clock.cc:648] HybridClock initialized: now 1790585924726858 us; error 43 us; skew 500 ppm
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:44.729225 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:58:44.729578 640 webserver.cc:460] Webserver started at http://127.0.98.2:46863/ using document root <none> and password file <none>
I20260928 08:58:44.729774 640 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:44.729813 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:44.729897 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:44.730877 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "1532da1ba87a4ba1af055be3e6606350"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "75f218851beb092b146b5c6316b8abbb"
server_key_iv: "a6f4ab453131834a64ad5d0a38240808"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.731165 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1532da1ba87a4ba1af055be3e6606350"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "75f218851beb092b146b5c6316b8abbb"
server_key_iv: "a6f4ab453131834a64ad5d0a38240808"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.732384 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:58:44.733124 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.733273 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.733354 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "1532da1ba87a4ba1af055be3e6606350"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "75f218851beb092b146b5c6316b8abbb"
server_key_iv: "a6f4ab453131834a64ad5d0a38240808"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.733407 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:44.747272 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:44.749114 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:44.749217 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:44.749456 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:44.749688 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:44.749717 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.749763 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:44.749790 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.756013 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33467
I20260928 08:58:44.756063 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33467 every 8 connection(s)
I20260928 08:58:44.756367 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:58:44.759976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 640
I20260928 08:58:44.760076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:58:44.760246 392 external_mini_cluster.cc:1428] Setting key 5fd832af31c123013e4176493c928191
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:44.766887 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.757038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50175 (local address 127.0.98.62:35787)
0928 08:58:44.757118 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:44.757120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:44.757310 (+ 190us) server_negotiation.cc:407] Connection header received
0928 08:58:44.757844 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:44.757846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:44.757865 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:44.757926 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:44.758905 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.759440 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.760649 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.760955 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.761031 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.763643 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:44.763645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:44.763645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:44.763653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:44.765415 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.765820 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.765821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.765822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.765887 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.766180 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.766181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.766181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.766334 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:44.766413 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:44.766693 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:58:44.766777 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:58:44.766968 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.757003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35787 (local address 127.0.98.2:50175)
0928 08:58:44.757188 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:44.757231 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:44.757674 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:44.758000 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:44.758004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:44.758609 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:44.758807 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:44.758811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.759514 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.759514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:44.760571 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:44.760573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.761050 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.761051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:44.761193 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.761692 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:44.761702 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:44.763554 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:44.765508 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:44.765514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:44.765515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:44.765746 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:44.765966 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:44.765967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:44.765968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:44.766106 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:44.766419 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:44.766425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:44.766558 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 08:58:44.766652 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:58:44.766775 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260928 08:58:44.767468 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35787
I20260928 08:58:44.767570 769 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:44.767745 769 heartbeater.cc:511] Master 127.0.98.62:35787 requested a full tablet report, sending...
I20260928 08:58:44.768178 444 ts_manager.cc:194] Registered new tserver with Master: 1532da1ba87a4ba1af055be3e6606350 (127.0.98.2:33467)
I20260928 08:58:44.768575 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50175
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:58:44.787418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35787
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:44.868440 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:44.868706 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:44.868755 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:44.869876 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:44.869917 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:44.869942 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:58:44.871371 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:44.871711 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:44.871964 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:44.876951 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:44.876976 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.876981 774 server_base.cc:1034] running on GCE node
W20260928 08:58:44.876952 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:44.877321 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:44.877507 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:44.878638 774 hybrid_clock.cc:648] HybridClock initialized: now 1790585924878625 us; error 29 us; skew 500 ppm
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:44.881121 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:58:44.881532 774 webserver.cc:460] Webserver started at http://127.0.98.3:46103/ using document root <none> and password file <none>
I20260928 08:58:44.881768 774 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:44.881811 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:44.881959 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:44.882987 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "01f8357e4879413da5e552dbc77796dd"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "ee17e88e7697361b55d0039dd02a2cc5"
server_key_iv: "7f7c760eccc623f44c73e030c7736753"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.883342 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01f8357e4879413da5e552dbc77796dd"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "ee17e88e7697361b55d0039dd02a2cc5"
server_key_iv: "7f7c760eccc623f44c73e030c7736753"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.884677 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:58:44.885357 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.885509 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:44.885588 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "01f8357e4879413da5e552dbc77796dd"
format_stamp: "Formatted at 2026-09-28 08:58:44 on dist-test-slave-286n"
server_key: "ee17e88e7697361b55d0039dd02a2cc5"
server_key_iv: "7f7c760eccc623f44c73e030c7736753"
server_key_version: "encryptionkey@0"
I20260928 08:58:44.885638 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:44.904206 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:44.906040 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:44.906147 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:44.906325 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:44.906553 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:44.906582 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.906602 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:44.906615 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:44.912881 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43271
I20260928 08:58:44.912937 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43271 every 8 connection(s)
I20260928 08:58:44.913260 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:44.922313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 774
I20260928 08:58:44.922420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:58:44.922580 392 external_mini_cluster.cc:1428] Setting key c43dc2a45cbd1c317ffa29b7fa0006ef
I20260928 08:58:44.923337 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.914942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40225 (local address 127.0.98.62:35787)
0928 08:58:44.914997 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:44.914999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:44.915206 (+ 207us) server_negotiation.cc:407] Connection header received
0928 08:58:44.915658 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:44.915659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:44.915667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:44.915724 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:44.916619 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.917165 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.918181 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.918391 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.918446 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.920491 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:44.920492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:44.920492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:44.920498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:44.921909 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.922332 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.922333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.922334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.922380 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.922649 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.922650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.922651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.922765 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:44.922835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:44.923158 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 08:58:44.923243 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 08:58:44.923565 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.914917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35787 (local address 127.0.98.3:40225)
0928 08:58:44.915089 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:44.915130 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:44.915582 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:44.915778 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:44.915781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:44.916352 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:44.916544 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:44.916549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.917234 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.917235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:44.918045 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:44.918046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.918457 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.918458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:44.918554 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.919015 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:44.919023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:44.920408 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:44.921975 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:44.921980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:44.921982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:44.922251 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:44.922467 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:44.922469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:44.922469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:44.922583 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:44.922827 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:44.922832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:44.923030 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 08:58:44.923140 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 08:58:44.923317 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 08:58:44.924000 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35787
I20260928 08:58:44.924086 903 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:44.924217 903 heartbeater.cc:511] Master 127.0.98.62:35787 requested a full tablet report, sending...
I20260928 08:58:44.924530 444 ts_manager.cc:194] Registered new tserver with Master: 01f8357e4879413da5e552dbc77796dd (127.0.98.3:43271)
I20260928 08:58:44.924903 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40225
I20260928 08:58:44.934551 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:58:44.957911 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.950040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.98.1:34153)
0928 08:58:44.950188 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:44.950193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:44.950204 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:58:44.950242 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:44.950244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:44.950258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:44.950322 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:44.950869 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.951650 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.952476 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.952852 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.952912 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.955248 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:44.955255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:44.955255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:44.955261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:44.956721 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.957077 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.957085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.957086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.957137 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.957339 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.957340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.957341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.957472 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:44.957611 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:44.957708 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 08:58:44.957790 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260928 08:58:44.958276 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43484
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:58:44 dist-test-slave-286n krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:58:44.980053 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:44.973184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.0.98.1:34153)
0928 08:58:44.973249 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:44.973251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:44.973353 (+ 102us) server_negotiation.cc:407] Connection header received
0928 08:58:44.973441 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:44.973442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:44.973461 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:44.973521 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:44.974082 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.974574 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.975313 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:44.975515 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:44.975599 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:44.977714 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:44.977715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:44.977716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:44.977724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:44.978965 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.979321 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.979322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.979322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.979375 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:44.979558 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:44.979558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:44.979559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:44.979656 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:44.979773 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:44.979872 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 08:58:44.979951 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 08:58:44.980520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 506
I20260928 08:58:44.985083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 640
I20260928 08:58:44.990447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 774
I20260928 08:58:44.994616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 412
2026-09-28T08:58:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4861 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:45 dist-test-slave-286n krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:45 dist-test-slave-286n krb5kdc[917](info): set up 2 sockets
Sep 28 08:58:45 dist-test-slave-286n krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 08:58:46.020059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:58:46.135991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46045
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:46.223402 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:46.223636 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:46.223675 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:46.224882 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:46.224941 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:46.224964 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:46.224992 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:46.225021 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:46.226504 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:46.226783 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:46.226998 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:46.232025 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.232100 930 server_base.cc:1034] running on GCE node
W20260928 08:58:46.232044 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:46.232049 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.232425 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:46.232785 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:46.233973 930 hybrid_clock.cc:648] HybridClock initialized: now 1790585926233960 us; error 38 us; skew 500 ppm
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:46.236495 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:46.237020 930 webserver.cc:460] Webserver started at http://127.0.98.62:44623/ using document root <none> and password file <none>
I20260928 08:58:46.237241 930 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:46.237304 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:46.237453 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:46.238507 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "1d2abe8adf0e48889f84a0626439a5e8"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "8853505d37ec4ced2c4d50f6bbc65b48"
server_key_iv: "39b03ae3eac7eb04b4d86ef4cbb9431a"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.238865 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d2abe8adf0e48889f84a0626439a5e8"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "8853505d37ec4ced2c4d50f6bbc65b48"
server_key_iv: "39b03ae3eac7eb04b4d86ef4cbb9431a"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.240155 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:46.241132 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.241303 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.241400 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "1d2abe8adf0e48889f84a0626439a5e8"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "8853505d37ec4ced2c4d50f6bbc65b48"
server_key_iv: "39b03ae3eac7eb04b4d86ef4cbb9431a"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.241473 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:46.254168 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:46.256419 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:46.256573 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:46.260362 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46045
I20260928 08:58:46.260417 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46045 every 8 connection(s)
I20260928 08:58:46.260845 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:46.261656 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:46.263805 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8: Bootstrap starting.
I20260928 08:58:46.264321 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:46.264555 998 log.cc:824] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:46.265225 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8: No bootstrap required, opened a new log
I20260928 08:58:46.266719 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2abe8adf0e48889f84a0626439a5e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46045 } }
I20260928 08:58:46.266851 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:46.266882 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2abe8adf0e48889f84a0626439a5e8, State: Initialized, Role: FOLLOWER
I20260928 08:58:46.266983 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2abe8adf0e48889f84a0626439a5e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46045 } }
I20260928 08:58:46.267056 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:46.267091 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:46.267159 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:46.267668 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2abe8adf0e48889f84a0626439a5e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46045 } }
I20260928 08:58:46.267777 998 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2abe8adf0e48889f84a0626439a5e8; no voters:
I20260928 08:58:46.267949 998 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:46.267992 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:46.268196 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:46.268429 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [term 1 LEADER]: Becoming Leader. State: Replica: 1d2abe8adf0e48889f84a0626439a5e8, State: Running, Role: LEADER
I20260928 08:58:46.268579 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2abe8adf0e48889f84a0626439a5e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46045 } }
I20260928 08:58:46.269006 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2abe8adf0e48889f84a0626439a5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2abe8adf0e48889f84a0626439a5e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46045 } } }
I20260928 08:58:46.269016 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2abe8adf0e48889f84a0626439a5e8. Latest consensus state: current_term: 1 leader_uuid: "1d2abe8adf0e48889f84a0626439a5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2abe8adf0e48889f84a0626439a5e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46045 } } }
I20260928 08:58:46.269090 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:46.269121 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:46.269291 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:46.269774 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:46.270833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 930
I20260928 08:58:46.270942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:46.271091 392 external_mini_cluster.cc:1428] Setting key a2797a771dc666c706677adc91ec7162
I20260928 08:58:46.273066 1010 catalog_manager.cc:1347] Generated new cluster ID: 8a5847e8ba7c438795aaa49826daff9b
I20260928 08:58:46.273115 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:58:46.276726 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:46.277483 1010 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:46.282310 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.271551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.98.62:46045)
0928 08:58:46.271720 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.271729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.271781 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:58:46.271896 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.271903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.271957 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.272098 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.272686 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.274108 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.275128 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.275758 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.275870 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.278664 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.278668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.278669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.278680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.280754 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.281105 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.281108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.281109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.281215 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.281411 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.281412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.281413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.281684 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.281834 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.281929 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.282077 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260928 08:58:46.283071 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2abe8adf0e48889f84a0626439a5e8: Generated new TSK 0
I20260928 08:58:46.283219 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:58:46.315390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46045
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:46.396486 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:46.396714 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:46.396740 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:46.397843 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:46.397881 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:46.397909 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:58:46.399237 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:46.399463 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:46.399642 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:46.404830 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:46.404840 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:46.404841 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.404989 1024 server_base.cc:1034] running on GCE node
I20260928 08:58:46.405328 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:46.405548 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:46.406684 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790585926406673 us; error 28 us; skew 500 ppm
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:46.409073 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:58:46.409487 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45761/ using document root <none> and password file <none>
I20260928 08:58:46.409706 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:46.409749 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:46.409835 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:46.410789 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "4533026f41584394b6b1262118c26050"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "ceb1657d88aceccb921cbb2d91a24262"
server_key_iv: "d8cc6892edfee45cec4a0c5d93e28b86"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.411115 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4533026f41584394b6b1262118c26050"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "ceb1657d88aceccb921cbb2d91a24262"
server_key_iv: "d8cc6892edfee45cec4a0c5d93e28b86"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.412262 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:46.412895 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.413044 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:46.413127 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "4533026f41584394b6b1262118c26050"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "ceb1657d88aceccb921cbb2d91a24262"
server_key_iv: "d8cc6892edfee45cec4a0c5d93e28b86"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.413192 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:46.436388 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:46.438381 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:46.438520 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:46.438746 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:46.439047 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:46.439096 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.439138 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:46.439164 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.445075 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41995
I20260928 08:58:46.445165 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41995 every 8 connection(s)
I20260928 08:58:46.445464 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:58:46.450179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1024
I20260928 08:58:46.450284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:58:46.450444 392 external_mini_cluster.cc:1428] Setting key e49b4f57a286c6e1b8369107bb886848
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:46.455449 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.446223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39957 (local address 127.0.98.62:46045)
0928 08:58:46.446294 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.446296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.446484 (+ 188us) server_negotiation.cc:407] Connection header received
0928 08:58:46.446989 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.446990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.447002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.447040 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.447985 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.448476 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.449449 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.449639 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.449688 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.452154 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.452156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.452156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.452164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.453976 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.454409 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.454411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.454411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.454481 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.454731 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.454732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.454732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.454868 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.454938 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.455194 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.455292 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:58:46.455471 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.446176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46045 (local address 127.0.98.1:39957)
0928 08:58:46.446355 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.446394 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.446892 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.447112 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.447116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.447695 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:46.447905 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.447910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.448554 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.448555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.449367 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.449370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.449712 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.449713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.449818 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.450447 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:46.450456 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:46.452038 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:46.454085 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:46.454091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:46.454092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:46.454322 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:46.454556 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:46.454556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:46.454557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:46.454664 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:46.454945 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:46.454950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:46.455067 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.455149 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.455304 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260928 08:58:46.456178 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46045
I20260928 08:58:46.456331 1153 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:46.456516 1153 heartbeater.cc:511] Master 127.0.98.62:46045 requested a full tablet report, sending...
I20260928 08:58:46.457065 962 ts_manager.cc:194] Registered new tserver with Master: 4533026f41584394b6b1262118c26050 (127.0.98.1:41995)
I20260928 08:58:46.457679 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39957
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:58:46.476310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46045
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:46.554876 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:46.555121 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:46.555169 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:46.556334 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:46.556399 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:46.556445 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:58:46.557845 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:46.558109 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:46.558319 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:46.563155 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.563311 1158 server_base.cc:1034] running on GCE node
W20260928 08:58:46.563248 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:46.563155 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.563625 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:46.563851 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:46.564981 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790585926564973 us; error 27 us; skew 500 ppm
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:46.567355 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:58:46.567708 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41407/ using document root <none> and password file <none>
I20260928 08:58:46.567878 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:46.567916 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:46.568001 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:46.569128 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "b3c56b3a57824fcaa6d13788e3d9512c"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "a3ba626a141abc81699b3402372c059e"
server_key_iv: "13b1769ce9c2df38bd7e166b3e4c3634"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.569454 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b3c56b3a57824fcaa6d13788e3d9512c"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "a3ba626a141abc81699b3402372c059e"
server_key_iv: "13b1769ce9c2df38bd7e166b3e4c3634"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.570782 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:46.571434 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.571604 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:46.571686 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "b3c56b3a57824fcaa6d13788e3d9512c"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "a3ba626a141abc81699b3402372c059e"
server_key_iv: "13b1769ce9c2df38bd7e166b3e4c3634"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.571756 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:46.586946 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:46.589196 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:46.589365 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:46.589602 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:46.589915 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:46.589974 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.590018 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:46.590044 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.595580 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42381
I20260928 08:58:46.595654 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42381 every 8 connection(s)
I20260928 08:58:46.595963 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:58:46.600881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1158
I20260928 08:58:46.600988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:58:46.601146 392 external_mini_cluster.cc:1428] Setting key 899048403e3096ab43b11e281d062fb4
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:46.605556 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.596674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51095 (local address 127.0.98.62:46045)
0928 08:58:46.596729 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.596731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.596943 (+ 212us) server_negotiation.cc:407] Connection header received
0928 08:58:46.597437 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.597438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.597453 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.597507 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.598421 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.598899 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.599862 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.600036 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.600084 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.602766 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.602767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.602768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.602774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.604195 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.604601 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.604603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.604603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.604686 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.604914 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.604915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.604916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.605035 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.605103 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.605378 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.605462 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:58:46.605760 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.596619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46045 (local address 127.0.98.2:51095)
0928 08:58:46.596829 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.596864 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.597345 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.597570 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.597574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.598147 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:46.598344 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.598348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.598972 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.598972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.599791 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.599793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.600100 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.600100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.600193 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.600744 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:46.600755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:46.602680 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:46.604285 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:46.604291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:46.604294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:46.604512 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:46.604736 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:46.604737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:46.604738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:46.604847 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:46.605108 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:46.605114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:46.605255 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.605359 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.605523 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260928 08:58:46.606273 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46045
I20260928 08:58:46.606369 1287 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:46.606565 1287 heartbeater.cc:511] Master 127.0.98.62:46045 requested a full tablet report, sending...
I20260928 08:58:46.606938 962 ts_manager.cc:194] Registered new tserver with Master: b3c56b3a57824fcaa6d13788e3d9512c (127.0.98.2:42381)
I20260928 08:58:46.607308 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51095
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:58:46.628160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46045
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:46.712162 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:46.712396 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:46.712437 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:46.713689 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:46.713768 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:46.713815 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:58:46.715375 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46045
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:46.715629 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:46.715909 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:46.721009 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:46.720974 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.721011 1292 server_base.cc:1034] running on GCE node
W20260928 08:58:46.720970 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:46.721429 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:46.721688 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:46.722856 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790585926722851 us; error 33 us; skew 500 ppm
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:46.725319 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:58:46.725745 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46283/ using document root <none> and password file <none>
I20260928 08:58:46.725942 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:46.725983 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:46.726068 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:46.727062 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4078d827db941cfa87aec8755055a5c"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "08593f04bc80f2798add3785fc24d044"
server_key_iv: "95fc03699c29451ac4af9e194d81a9f7"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.727372 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4078d827db941cfa87aec8755055a5c"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "08593f04bc80f2798add3785fc24d044"
server_key_iv: "95fc03699c29451ac4af9e194d81a9f7"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.728575 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:46.729264 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.729408 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:46.729496 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "d4078d827db941cfa87aec8755055a5c"
format_stamp: "Formatted at 2026-09-28 08:58:46 on dist-test-slave-286n"
server_key: "08593f04bc80f2798add3785fc24d044"
server_key_iv: "95fc03699c29451ac4af9e194d81a9f7"
server_key_version: "encryptionkey@0"
I20260928 08:58:46.729549 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:46.758780 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:46.760756 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:46.760864 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:46.761058 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:46.761371 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:46.761404 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.761444 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:46.761472 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:46.766937 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37027
I20260928 08:58:46.767016 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37027 every 8 connection(s)
I20260928 08:58:46.767316 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 08:58:46.772929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1292
I20260928 08:58:46.773042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:58:46.773212 392 external_mini_cluster.cc:1428] Setting key 2273152e96aad853a0f71dafd60efa6e
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:46.776518 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.768063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51239 (local address 127.0.98.62:46045)
0928 08:58:46.768116 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.768119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.768418 (+ 299us) server_negotiation.cc:407] Connection header received
0928 08:58:46.768948 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.768949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.768959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.769049 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.769998 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.770531 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.771517 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.771723 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.771789 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.774028 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.774029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.774030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.774037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.775203 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.775601 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.775603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.775603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.775659 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.775900 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.775901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.775901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.776014 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.776083 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.776347 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.776425 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:58:46.776736 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.768042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46045 (local address 127.0.98.3:51239)
0928 08:58:46.768210 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.768246 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.768790 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.769108 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.769112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.769697 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:46.769895 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.769899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.770609 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.770610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.771418 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.771420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.771808 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.771808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.771901 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.772330 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:46.772338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:46.773926 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:46.775292 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:46.775297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:46.775299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:46.775529 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:46.775727 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:46.775728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:46.775728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:46.775836 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:46.776074 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:46.776080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:46.776214 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.776325 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.776486 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260928 08:58:46.777158 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46045
I20260928 08:58:46.777240 1421 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:46.777386 1421 heartbeater.cc:511] Master 127.0.98.62:46045 requested a full tablet report, sending...
I20260928 08:58:46.777693 962 ts_manager.cc:194] Registered new tserver with Master: d4078d827db941cfa87aec8755055a5c (127.0.98.3:37027)
I20260928 08:58:46.778004 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51239
I20260928 08:58:46.785187 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:58:46.791961 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.786616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.98.62:46045)
0928 08:58:46.786660 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.786661 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.786737 (+ 76us) server_negotiation.cc:407] Connection header received
0928 08:58:46.786789 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.786789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.786797 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.786853 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.787364 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.787828 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.788492 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.788712 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.788762 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.789921 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.789922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.789922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.789927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.791091 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.791393 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.791394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.791395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.791447 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.791590 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.791591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.791591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.791711 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.791782 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.791846 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.791902 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:58:46.793944 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:46.795506 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:46.802670 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.797999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41995 (local address 127.0.0.1:56692)
0928 08:58:46.798160 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.798199 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.798260 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.798728 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.798731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.798733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:46.798800 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.798801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.800146 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.800147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.801524 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.801528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.802457 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.802458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.802512 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.802523 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.802579 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.802615 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":54,"thread_start_us":46,"threads_started":1}
I20260928 08:58:46.802776 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.798128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.0.98.1:41995)
0928 08:58:46.798554 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.798559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.798569 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:58:46.798616 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.798618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.798631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.798705 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:46.798937 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.800039 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.801630 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.802371 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.802455 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.802515 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.802609 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.802652 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":89,"threads_started":1}
I20260928 08:58:46.803476 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.797999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42381 (local address 127.0.0.1:32946)
0928 08:58:46.798341 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.798354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.798416 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.799379 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.799380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.799383 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:46.799435 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.799437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.800555 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.800556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.801800 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.801802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.803248 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.803249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.803311 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.803315 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.803374 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.803411 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":16,"thread_start_us":74,"threads_started":1}
I20260928 08:58:46.803521 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.798128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.98.2:42381)
0928 08:58:46.798853 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.798858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.798868 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:58:46.798910 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.798912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.798928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.799382 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:46.799501 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.800407 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.801887 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.802678 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.802750 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.802804 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.803392 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.803436 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":85,"threads_started":1}
I20260928 08:58:46.803684 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9 (DEFAULT_TABLE table=test-table [id=84a17e87b2c84e4cb105b404788616bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:46.804039 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:46.804281 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.798186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37027 (local address 127.0.0.1:50794)
0928 08:58:46.799028 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.799040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.799088 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.799281 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.799282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.799284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:46.799322 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.799323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.800470 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.800471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.801799 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.801802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.804017 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.804018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.804102 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.804107 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.804162 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.804207 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260928 08:58:46.804469 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9 (DEFAULT_TABLE table=test-table [id=84a17e87b2c84e4cb105b404788616bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:46.804610 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.798393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.3:37027)
0928 08:58:46.798558 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.798561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.799160 (+ 599us) server_negotiation.cc:407] Connection header received
0928 08:58:46.799195 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.799197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.799209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.799274 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:46.799406 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.800370 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.802844 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.803855 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.803920 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.803982 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.804481 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.804529 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260928 08:58:46.804824 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:46.805187 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9 (DEFAULT_TABLE table=test-table [id=84a17e87b2c84e4cb105b404788616bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:46.805524 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:46.807523 1438 tablet_bootstrap.cc:492] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: Bootstrap starting.
I20260928 08:58:46.808117 1439 tablet_bootstrap.cc:492] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: Bootstrap starting.
I20260928 08:58:46.808781 1439 tablet_bootstrap.cc:654] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:46.808894 1438 tablet_bootstrap.cc:654] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:46.809039 1440 tablet_bootstrap.cc:492] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c: Bootstrap starting.
I20260928 08:58:46.809459 1439 log.cc:824] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:46.809497 1438 log.cc:824] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:46.809686 1440 tablet_bootstrap.cc:654] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:46.810250 1440 log.cc:824] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:46.810348 1439 tablet_bootstrap.cc:492] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: No bootstrap required, opened a new log
I20260928 08:58:46.810431 1438 tablet_bootstrap.cc:492] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: No bootstrap required, opened a new log
I20260928 08:58:46.810427 1439 ts_tablet_manager.cc:1397] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:58:46.810554 1438 ts_tablet_manager.cc:1397] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:58:46.810912 1440 tablet_bootstrap.cc:492] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c: No bootstrap required, opened a new log
I20260928 08:58:46.810997 1440 ts_tablet_manager.cc:1397] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:58:46.812469 1440 raft_consensus.cc:348] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.812440 1438 raft_consensus.cc:348] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.812440 1439 raft_consensus.cc:348] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.812623 1440 raft_consensus.cc:374] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:46.812621 1438 raft_consensus.cc:374] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:46.812621 1439 raft_consensus.cc:374] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:46.812695 1438 raft_consensus.cc:729] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4533026f41584394b6b1262118c26050, State: Initialized, Role: FOLLOWER
I20260928 08:58:46.812700 1439 raft_consensus.cc:729] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c56b3a57824fcaa6d13788e3d9512c, State: Initialized, Role: FOLLOWER
I20260928 08:58:46.812713 1440 raft_consensus.cc:729] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4078d827db941cfa87aec8755055a5c, State: Initialized, Role: FOLLOWER
I20260928 08:58:46.812824 1439 consensus_queue.cc:260] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.812861 1438 consensus_queue.cc:260] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.812903 1440 consensus_queue.cc:260] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.813117 1439 ts_tablet_manager.cc:1428] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 08:58:46.813158 1438 ts_tablet_manager.cc:1428] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 08:58:46.813184 1153 heartbeater.cc:503] Master 127.0.98.62:46045 was elected leader, sending a full tablet report...
I20260928 08:58:46.813648 1287 heartbeater.cc:503] Master 127.0.98.62:46045 was elected leader, sending a full tablet report...
I20260928 08:58:46.813897 1440 ts_tablet_manager.cc:1428] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:58:46.814033 1421 heartbeater.cc:503] Master 127.0.98.62:46045 was elected leader, sending a full tablet report...
W20260928 08:58:46.846482 1288 tablet.cc:2369] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:58:46.870309 1446 raft_consensus.cc:482] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:46.870438 1446 raft_consensus.cc:504] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.870756 1446 leader_election.cc:288] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4533026f41584394b6b1262118c26050 (127.0.98.1:41995), b3c56b3a57824fcaa6d13788e3d9512c (127.0.98.2:42381)
I20260928 08:58:46.873894 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.871033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49773 (local address 127.0.98.1:41995)
0928 08:58:46.871076 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.871079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.871090 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:58:46.871137 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.871138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.871150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.871244 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:46.871441 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.871960 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.873071 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.873527 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.873608 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.873619 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.873771 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.873817 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:58:46.873911 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.870957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41995 (local address 127.0.98.3:49773)
0928 08:58:46.871022 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.871035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.871078 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.871260 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.871262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.871263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:46.871361 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.871363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.872040 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.872041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.872979 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.872981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.873607 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.873607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.873684 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.873692 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.873765 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.873803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 08:58:46.873917 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.871033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56165 (local address 127.0.98.2:42381)
0928 08:58:46.871076 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.871079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.871248 (+ 169us) server_negotiation.cc:407] Connection header received
0928 08:58:46.871292 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.871293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.871311 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.871363 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:46.871550 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.872012 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.873075 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.873568 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.873645 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.873654 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.873799 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.873838 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:58:46.873930 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.870957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42381 (local address 127.0.98.3:56165)
0928 08:58:46.871142 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:46.871153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:46.871235 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:46.871374 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:46.871375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:46.871376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:46.871468 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.871469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.872097 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.872098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.873000 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:46.873003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.873644 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.873645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:46.873717 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.873720 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:58:46.873792 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 08:58:46.873830 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":20,"thread_start_us":109,"threads_started":1}
I20260928 08:58:46.874445 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb863e3d03451eb86a6fc2d7a0f3e9" candidate_uuid: "d4078d827db941cfa87aec8755055a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4533026f41584394b6b1262118c26050" is_pre_election: true
I20260928 08:58:46.874606 1107 raft_consensus.cc:2454] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4078d827db941cfa87aec8755055a5c in term 0.
I20260928 08:58:46.874584 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb863e3d03451eb86a6fc2d7a0f3e9" candidate_uuid: "d4078d827db941cfa87aec8755055a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" is_pre_election: true
I20260928 08:58:46.874720 1241 raft_consensus.cc:2454] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4078d827db941cfa87aec8755055a5c in term 0.
I20260928 08:58:46.874840 1309 leader_election.cc:302] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4533026f41584394b6b1262118c26050, d4078d827db941cfa87aec8755055a5c; no voters:
I20260928 08:58:46.874969 1446 raft_consensus.cc:2781] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:46.875036 1446 raft_consensus.cc:482] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:46.875067 1446 raft_consensus.cc:3037] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:46.875850 1446 raft_consensus.cc:504] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.875983 1446 leader_election.cc:288] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [CANDIDATE]: Term 1 election: Requested vote from peers 4533026f41584394b6b1262118c26050 (127.0.98.1:41995), b3c56b3a57824fcaa6d13788e3d9512c (127.0.98.2:42381)
I20260928 08:58:46.876142 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb863e3d03451eb86a6fc2d7a0f3e9" candidate_uuid: "d4078d827db941cfa87aec8755055a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c56b3a57824fcaa6d13788e3d9512c"
I20260928 08:58:46.876197 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbb863e3d03451eb86a6fc2d7a0f3e9" candidate_uuid: "d4078d827db941cfa87aec8755055a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4533026f41584394b6b1262118c26050"
I20260928 08:58:46.876226 1241 raft_consensus.cc:3037] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:46.876279 1107 raft_consensus.cc:3037] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:46.877043 1241 raft_consensus.cc:2454] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P b3c56b3a57824fcaa6d13788e3d9512c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4078d827db941cfa87aec8755055a5c in term 1.
I20260928 08:58:46.877065 1107 raft_consensus.cc:2454] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4078d827db941cfa87aec8755055a5c in term 1.
I20260928 08:58:46.877233 1308 leader_election.cc:302] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c56b3a57824fcaa6d13788e3d9512c, d4078d827db941cfa87aec8755055a5c; no voters:
I20260928 08:58:46.877357 1446 raft_consensus.cc:2781] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:46.877521 1446 raft_consensus.cc:686] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [term 1 LEADER]: Becoming Leader. State: Replica: d4078d827db941cfa87aec8755055a5c, State: Running, Role: LEADER
I20260928 08:58:46.877717 1446 consensus_queue.cc:237] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } }
I20260928 08:58:46.878522 961 catalog_manager.cc:5495] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P d4078d827db941cfa87aec8755055a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to d4078d827db941cfa87aec8755055a5c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d4078d827db941cfa87aec8755055a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4533026f41584394b6b1262118c26050" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3c56b3a57824fcaa6d13788e3d9512c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4078d827db941cfa87aec8755055a5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37027 } health_report { overall_health: HEALTHY } } }
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:58:46.900750 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.892948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.98.1:41995)
0928 08:58:46.893011 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.893013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.893578 (+ 565us) server_negotiation.cc:407] Connection header received
0928 08:58:46.893663 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.893664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.893677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.893744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.894336 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.894821 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.895544 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.895740 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.895820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.898108 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.898121 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.898121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.898128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.899558 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.899914 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.899918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.899919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.899983 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.900171 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.900172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.900173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.900315 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.900493 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.900529 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.900608 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:58:46.923566 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.916030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.98.1:41995)
0928 08:58:46.916071 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.916072 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.916224 (+ 152us) server_negotiation.cc:407] Connection header received
0928 08:58:46.916306 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.916307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.916360 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.916426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.917062 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.917536 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.918523 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.918721 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.918822 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.921070 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.921072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.921072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.921080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.922361 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.922726 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.922727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.922728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.922786 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.922998 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.922999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.922999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.923098 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.923195 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.923386 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.923462 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:58:46 dist-test-slave-286n krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:58:46.945079 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:46.938395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.0.98.1:41995)
0928 08:58:46.938458 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:46.938460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:46.938473 (+ 13us) server_negotiation.cc:407] Connection header received
0928 08:58:46.938498 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:46.938499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:46.938508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:46.938559 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:46.939097 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.939548 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.940251 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:46.940419 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:46.940466 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:46.942676 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:46.942677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:46.942677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:46.942683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:46.944089 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.944441 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.944442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.944442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.944493 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:46.944691 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:46.944692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:46.944692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:46.944790 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:46.944859 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:46.944960 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 08:58:46.945013 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 08:58:46.945622 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0dbb863e3d03451eb86a6fc2d7a0f3e9: Uninitialized: clean time has not yet been initialized
W20260928 08:58:46.946110 1154 tablet.cc:2369] T 0dbb863e3d03451eb86a6fc2d7a0f3e9 P 4533026f41584394b6b1262118c26050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:58:46.946529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1024
I20260928 08:58:46.951623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1158
I20260928 08:58:46.956128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1292
I20260928 08:58:46.960315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 930
2026-09-28T08:58:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1966 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:47 dist-test-slave-286n krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:47 dist-test-slave-286n krb5kdc[1462](info): set up 2 sockets
Sep 28 08:58:47 dist-test-slave-286n krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260928 08:58:48.012517 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:58:48.133545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:48.217545 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:48.217751 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:48.217773 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:48.218914 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:48.218953 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:58:48.218966 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:48.218981 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:48.218992 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:48.219040 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:48.220495 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38157
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:48.220759 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:48.220990 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:48.225719 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:48.225786 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:48.225817 1475 server_base.cc:1034] running on GCE node
W20260928 08:58:48.225739 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:48.226306 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:48.226583 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:48.227738 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790585928227712 us; error 43 us; skew 500 ppm
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:48.230569 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:48.230978 1475 webserver.cc:460] Webserver started at http://127.0.98.62:38691/ using document root <none> and password file <none>
I20260928 08:58:48.231184 1475 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:48.231230 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:48.231333 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:48.232425 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "963a62cdbe8646b180035e27fa172721"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "a5dcbf08aac73266ce3bc6918069385c"
server_key_iv: "85ba54769272229c5bdf13aaffa0344a"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.232810 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "963a62cdbe8646b180035e27fa172721"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "a5dcbf08aac73266ce3bc6918069385c"
server_key_iv: "85ba54769272229c5bdf13aaffa0344a"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.234166 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:58:48.234892 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.235054 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:48.235160 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "963a62cdbe8646b180035e27fa172721"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "a5dcbf08aac73266ce3bc6918069385c"
server_key_iv: "85ba54769272229c5bdf13aaffa0344a"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.235232 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:48.252125 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:48.254505 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:48.254664 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:48.259292 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38157
I20260928 08:58:48.259356 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38157 every 8 connection(s)
I20260928 08:58:48.259704 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:48.260385 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.262724 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: Bootstrap starting.
I20260928 08:58:48.263338 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.263646 1553 log.cc:824] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:48.264340 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: No bootstrap required, opened a new log
I20260928 08:58:48.266718 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963a62cdbe8646b180035e27fa172721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260928 08:58:48.266878 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.266903 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963a62cdbe8646b180035e27fa172721, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.267107 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963a62cdbe8646b180035e27fa172721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260928 08:58:48.267181 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:48.267235 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:48.267292 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.267937 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963a62cdbe8646b180035e27fa172721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260928 08:58:48.268033 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963a62cdbe8646b180035e27fa172721; no voters:
I20260928 08:58:48.268280 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:48.268316 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:48.268313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1475
I20260928 08:58:48.268437 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [term 1 LEADER]: Becoming Leader. State: Replica: 963a62cdbe8646b180035e27fa172721, State: Running, Role: LEADER
I20260928 08:58:48.268433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:48.268517 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963a62cdbe8646b180035e27fa172721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260928 08:58:48.268602 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:48.268669 392 external_mini_cluster.cc:1428] Setting key 8ff6952280ed184ce411ecbbaa431276
I20260928 08:58:48.268909 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963a62cdbe8646b180035e27fa172721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963a62cdbe8646b180035e27fa172721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } } }
I20260928 08:58:48.269058 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:48.268893 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963a62cdbe8646b180035e27fa172721. Latest consensus state: current_term: 1 leader_uuid: "963a62cdbe8646b180035e27fa172721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963a62cdbe8646b180035e27fa172721" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } } }
I20260928 08:58:48.269446 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:48.269578 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:48.270097 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:48.271649 1562 catalog_manager.cc:1347] Generated new cluster ID: 3559fa889bd646e1bb8f5a75cfca9f9c
I20260928 08:58:48.271699 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:48.280018 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.269323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.0.98.62:38157)
0928 08:58:48.270170 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.270181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.270219 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:58:48.270334 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.270340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.270386 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.270545 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.271379 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.272493 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.273396 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.273878 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.274020 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.276480 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.276484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.276485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.276497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.278316 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.278720 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.278724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.278724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.278820 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.279040 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.279041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.279042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.279303 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.279493 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.279570 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.279842 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":68,"threads_started":1}
I20260928 08:58:48.298934 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:48.299489 1562 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:58:48.308710 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: Generated new TSK 0
I20260928 08:58:48.308914 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:58:48.341603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:48.447132 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:48.447333 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:48.447356 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:48.448503 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:48.448544 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:48.448565 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:58:48.448626 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:58:48.450196 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:48.450434 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:48.450655 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:48.455526 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:48.455638 1579 server_base.cc:1034] running on GCE node
W20260928 08:58:48.455559 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:48.455745 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:48.455947 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:48.456177 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:48.457329 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790585928457323 us; error 35 us; skew 500 ppm
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:48.460403 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:58:48.460901 1579 webserver.cc:460] Webserver started at http://127.0.98.1:38609/ using document root <none> and password file <none>
I20260928 08:58:48.461144 1579 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:48.461206 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:48.461360 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:48.462404 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "231aff02cbce456ca83bfa43440b1a17"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "33625831c61366ff81b443294c67312d"
server_key_iv: "910edf97eff785b26afec1a0418cda50"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.462790 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "231aff02cbce456ca83bfa43440b1a17"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "33625831c61366ff81b443294c67312d"
server_key_iv: "910edf97eff785b26afec1a0418cda50"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.464340 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:48.465126 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.465286 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.465377 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "231aff02cbce456ca83bfa43440b1a17"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "33625831c61366ff81b443294c67312d"
server_key_iv: "910edf97eff785b26afec1a0418cda50"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.465452 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:48.481743 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:48.483649 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:48.483798 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:48.484256 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:48.484313 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.484359 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:48.484403 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:48.491335 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35049
I20260928 08:58:48.491403 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35049 every 8 connection(s)
I20260928 08:58:48.491730 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:58:48.494244 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.484799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39541 (local address 127.0.98.62:38157)
0928 08:58:48.484848 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.484850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.485089 (+ 239us) server_negotiation.cc:407] Connection header received
0928 08:58:48.485611 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.485612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.485628 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.485690 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.486441 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.487072 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.488625 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.489019 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.489133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.491687 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.491688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.491689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.491699 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.493151 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.493497 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.493498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.493498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.493544 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.493747 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.493747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.493748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.493841 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.493890 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.494114 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.494175 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:58:48.494397 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.484800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.1:39541)
0928 08:58:48.484947 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.484980 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.485505 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.485760 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.485764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.486135 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.486338 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.486343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.487165 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.487166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.488516 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.488519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.489178 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.489178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.489294 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.490135 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.490143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.491593 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.493216 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.493222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.493223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.493433 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.493598 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.493599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.493599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.493696 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.493891 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.493894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.494014 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.494093 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.494216 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 08:58:48.494868 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38157
I20260928 08:58:48.494949 1710 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:48.495117 1710 heartbeater.cc:511] Master 127.0.98.62:38157 requested a full tablet report, sending...
I20260928 08:58:48.495651 1506 ts_manager.cc:194] Registered new tserver with Master: 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049)
I20260928 08:58:48.496322 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39541
I20260928 08:58:48.496942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1579
I20260928 08:58:48.497042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:58:48.497201 392 external_mini_cluster.cc:1428] Setting key 1948721bec394cd5ab9e6903664d1b07
I20260928 08:58:48.501890 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.495817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.1:51839)
0928 08:58:48.495937 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.495948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.496021 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.496147 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.496148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.496324 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.496401 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.496403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.497332 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.497333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.498048 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.498050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.498436 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.498437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.498499 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.498915 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.498918 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.499174 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.500974 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.500974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.500975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.501140 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.501328 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.501328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.501329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.501376 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.501645 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.501645 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.501681 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.501730 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.501819 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260928 08:58:48.501955 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.495849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51839 (local address 127.0.98.62:38157)
0928 08:58:48.495886 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.495888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.496007 (+ 119us) server_negotiation.cc:407] Connection header received
0928 08:58:48.496077 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.496078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.496088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.496143 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.496473 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.497207 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.498157 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.498357 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.498435 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.499268 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.499269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.499269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.499276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.500876 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.501204 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.501205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.501205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.501265 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.501431 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.501432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.501432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.501571 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.501638 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.501773 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.501857 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:58:48.522845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:48.601387 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:48.601686 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:48.601763 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:48.602957 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:48.603041 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:48.603080 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:58:48.603132 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:58:48.605151 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:48.605423 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:48.605721 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:48.610549 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:48.610589 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:48.610733 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:48.610818 1719 server_base.cc:1034] running on GCE node
I20260928 08:58:48.611011 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:48.611241 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:48.612401 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790585928612392 us; error 38 us; skew 500 ppm
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:48.614944 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:58:48.615406 1719 webserver.cc:460] Webserver started at http://127.0.98.2:33541/ using document root <none> and password file <none>
I20260928 08:58:48.615669 1719 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:48.615733 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:48.615875 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:48.616966 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba7472cae13746a399036a92785d5a07"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "5bc9f309fbb114d158028e24da6ccc84"
server_key_iv: "85fb277d18dcf0be7ee237031621a427"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.617305 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba7472cae13746a399036a92785d5a07"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "5bc9f309fbb114d158028e24da6ccc84"
server_key_iv: "85fb277d18dcf0be7ee237031621a427"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.618536 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:48.619203 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.619355 1719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:48.619447 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "ba7472cae13746a399036a92785d5a07"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "5bc9f309fbb114d158028e24da6ccc84"
server_key_iv: "85fb277d18dcf0be7ee237031621a427"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.619513 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:48.635196 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:48.637095 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:48.637248 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:48.637775 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:48.637833 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.637876 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:48.637902 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:48.644917 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35541
I20260928 08:58:48.645321 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:58:48.645303 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35541 every 8 connection(s)
I20260928 08:58:48.647208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1719
I20260928 08:58:48.647243 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.638182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38671 (local address 127.0.98.62:38157)
0928 08:58:48.638247 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.638249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.638440 (+ 191us) server_negotiation.cc:407] Connection header received
0928 08:58:48.639006 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.639007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.639020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.639103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.639816 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.640536 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.641550 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.641754 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.641835 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.644396 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.644398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.644398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.644406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.646028 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.646411 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.646412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.646412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.646470 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.646701 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.646702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.646702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.646806 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.646859 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.647111 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.647180 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:58:48.647307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:58:48.647478 392 external_mini_cluster.cc:1428] Setting key 71e3d923d19b3efb7228a40ef046e6ae
I20260928 08:58:48.647481 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.638178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.2:38671)
0928 08:58:48.638315 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.638353 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.638910 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.639197 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.639201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.639531 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.639725 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.639729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.640625 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.640626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.641449 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.641451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.641843 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.641844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.641967 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.642455 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.642463 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.644298 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.646104 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.646108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.646109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.646326 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.646556 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.646557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.646557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.646646 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.646857 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.646861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.646965 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.647068 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.647254 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260928 08:58:48.648064 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38157
I20260928 08:58:48.648152 1850 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:48.648324 1850 heartbeater.cc:511] Master 127.0.98.62:38157 requested a full tablet report, sending...
I20260928 08:58:48.648857 1506 ts_manager.cc:194] Registered new tserver with Master: ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541)
I20260928 08:58:48.649363 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38671
I20260928 08:58:48.655158 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.649010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.2:44287)
0928 08:58:48.649138 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.649147 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.649199 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.649317 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.649318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.649524 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.649589 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.649591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.650598 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.650599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.651096 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.651097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.651574 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.651575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.651623 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.652065 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.652070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.652357 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.654170 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.654171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.654171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.654341 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.654545 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.654546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.654546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.654597 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.654922 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.654922 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.654954 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.655019 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.655090 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260928 08:58:48.655259 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.649050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44287 (local address 127.0.98.62:38157)
0928 08:58:48.649088 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.649090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.649203 (+ 113us) server_negotiation.cc:407] Connection header received
0928 08:58:48.649247 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.649247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.649258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.649316 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.649677 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.650151 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.651174 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.651493 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.651563 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.652440 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.652441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.652442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.652449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.654077 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.654414 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.654416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.654416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.654474 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.654656 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.654657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.654657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.654855 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.654933 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.655063 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.655138 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:58:48.673362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:48.755002 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:48.755192 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:48.755215 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:48.756291 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:48.756323 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:48.756343 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:58:48.756361 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:58:48.757910 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:48.758087 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:48.758250 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:48.763120 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:48.763098 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:48.763108 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:48.763355 1859 server_base.cc:1034] running on GCE node
I20260928 08:58:48.763552 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:48.763768 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:48.764899 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790585928764890 us; error 28 us; skew 500 ppm
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:48.767233 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:58:48.767693 1859 webserver.cc:460] Webserver started at http://127.0.98.3:38817/ using document root <none> and password file <none>
I20260928 08:58:48.767890 1859 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:48.767930 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:48.768148 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:48.769424 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "479fc3259b584e84882ac78d0fe6f236"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "387aa466085f771d2069d36a860ef870"
server_key_iv: "455e66f6f97ff05f8b32fc50d3ce7e8b"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.769790 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "479fc3259b584e84882ac78d0fe6f236"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "387aa466085f771d2069d36a860ef870"
server_key_iv: "455e66f6f97ff05f8b32fc50d3ce7e8b"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.771219 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:48.771968 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.772138 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:48.772212 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "479fc3259b584e84882ac78d0fe6f236"
format_stamp: "Formatted at 2026-09-28 08:58:48 on dist-test-slave-286n"
server_key: "387aa466085f771d2069d36a860ef870"
server_key_iv: "455e66f6f97ff05f8b32fc50d3ce7e8b"
server_key_version: "encryptionkey@0"
I20260928 08:58:48.772264 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:48.780489 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:48.782689 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:48.782793 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:48.783208 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:48.783249 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.783272 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:48.783324 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.789633 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41847
I20260928 08:58:48.789686 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41847 every 8 connection(s)
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:48.789978 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 08:58:48.792753 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.783682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38555 (local address 127.0.98.62:38157)
0928 08:58:48.783749 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.783752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.783900 (+ 148us) server_negotiation.cc:407] Connection header received
0928 08:58:48.784372 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.784374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.784387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.784474 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.785317 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.786143 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.787680 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.787988 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.788088 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.790511 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.790512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.790513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.790519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.791648 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.791981 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.791981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.791982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.792042 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.792243 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.792244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.792244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.792337 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.792386 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.792576 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.792664 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:58:48.792943 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.783641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.3:38555)
0928 08:58:48.783778 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.783816 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.784275 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.784543 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.784547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.784943 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.785213 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.785219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.786238 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.786239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.787569 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.787571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.788089 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.788090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.788188 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.788728 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.788739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.790430 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.791713 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.791718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.791719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.791920 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.792096 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.792097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.792097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.792194 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.792385 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.792389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.792487 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.792561 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.792713 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260928 08:58:48.793510 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38157
I20260928 08:58:48.793596 1990 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:48.793772 1990 heartbeater.cc:511] Master 127.0.98.62:38157 requested a full tablet report, sending...
I20260928 08:58:48.794106 1506 ts_manager.cc:194] Registered new tserver with Master: 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3:41847)
I20260928 08:58:48.794456 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38555
I20260928 08:58:48.797955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 1859
I20260928 08:58:48.798058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:58:48.798224 392 external_mini_cluster.cc:1428] Setting key 12508e4c22755d370a43f940ac24d25a
I20260928 08:58:48.798966 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:58:48.799458 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.794553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45273 (local address 127.0.98.62:38157)
0928 08:58:48.794586 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.794589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.794734 (+ 145us) server_negotiation.cc:407] Connection header received
0928 08:58:48.794799 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.794799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.794813 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.794851 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.795171 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.795597 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.796206 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.796370 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.796418 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.797157 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.797159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.797159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.797165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.798334 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.798721 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.798722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.798722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.798781 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.798963 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.798963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.798964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.799087 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.799149 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.799305 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.799369 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:58:48.799472 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.794484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.3:45273)
0928 08:58:48.794655 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.794673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.794725 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.794857 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.794858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.795051 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.795103 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.795105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.795678 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.795679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.796141 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.796142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.796427 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.796428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.796471 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.796844 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.796848 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.797090 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.798425 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.798426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.798426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.798640 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.798841 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.798842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.798842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.798898 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.799159 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.799160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.799198 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.799270 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.799370 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:48.830142 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.822395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.0.98.62:38157)
0928 08:58:48.822462 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.822464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.822473 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:48.822536 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.822538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.822550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.822631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.823392 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.824085 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.825134 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.825352 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.825433 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.827649 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.827650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.827650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.827657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.829250 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.829558 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.829558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.829559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.829608 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.829748 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.829748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.829748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.829840 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.829910 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.829984 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.830040 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:58:48.831858 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:48.833464 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:48.839905 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.836031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.0.98.1:35049)
0928 08:58:48.836184 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.836188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.836202 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:58:48.836232 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.836234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.836245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.836334 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:48.836474 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.837259 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.838725 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.839483 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.839578 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.839659 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.839760 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.839804 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260928 08:58:48.839897 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.835816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.0.1:59392)
0928 08:58:48.836027 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.836038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.836082 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.836335 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.836336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.836338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:48.836391 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.836393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.837353 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.837354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.838617 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.838619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.839576 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.839577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.839654 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.839669 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.839740 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.839804 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260928 08:58:48.839953 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.835857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.0.98.2:35541)
0928 08:58:48.836005 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.836011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.836023 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:58:48.836086 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.836089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.836106 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.836181 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:48.836414 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.837569 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.838957 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.839489 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.839576 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.839659 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.839800 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.839848 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 08:58:48.840152 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.835635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35541 (local address 127.0.0.1:39756)
0928 08:58:48.835920 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.835931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.835983 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.836258 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.836261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.836263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:48.836332 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.836335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.837667 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.837668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.838862 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.838865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.839580 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.839582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.839683 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.839689 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.839746 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.840078 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":317,"thread_start_us":88,"threads_started":1}
I20260928 08:58:48.841089 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.835623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41847 (local address 127.0.0.1:42754)
0928 08:58:48.835736 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.835780 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.835846 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.836470 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.836471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.836473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:48.836510 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.836511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.837788 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.837789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.838891 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.838892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.840879 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.840880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.840950 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.840954 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.840986 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.841037 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260928 08:58:48.841092 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 9baefe10ee764b029bba89b00618519e (DEFAULT_TABLE table=test-table [id=b0701a6267d642d9859be720e9c39c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:48.841091 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 9baefe10ee764b029bba89b00618519e (DEFAULT_TABLE table=test-table [id=b0701a6267d642d9859be720e9c39c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:48.841447 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9baefe10ee764b029bba89b00618519e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.841447 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9baefe10ee764b029bba89b00618519e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.841600 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.835722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.0.98.3:41847)
0928 08:58:48.836192 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.836196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.836205 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:48.836242 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.836244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.836259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.836589 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:48.836676 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.837701 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.839699 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.840777 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.841298 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.841392 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.841420 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.841460 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":74,"threads_started":1}
I20260928 08:58:48.842161 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 9baefe10ee764b029bba89b00618519e (DEFAULT_TABLE table=test-table [id=b0701a6267d642d9859be720e9c39c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:48.842504 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9baefe10ee764b029bba89b00618519e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.845314 2013 tablet_bootstrap.cc:492] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Bootstrap starting.
I20260928 08:58:48.845551 2014 tablet_bootstrap.cc:492] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Bootstrap starting.
I20260928 08:58:48.845602 2015 tablet_bootstrap.cc:492] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Bootstrap starting.
I20260928 08:58:48.846131 2015 tablet_bootstrap.cc:654] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.846149 2014 tablet_bootstrap.cc:654] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.846432 2015 log.cc:824] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:48.846463 2014 log.cc:824] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:48.847229 2015 tablet_bootstrap.cc:492] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: No bootstrap required, opened a new log
I20260928 08:58:48.847231 2013 tablet_bootstrap.cc:654] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.847306 2015 ts_tablet_manager.cc:1397] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:48.847617 2013 log.cc:824] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:48.848205 2014 tablet_bootstrap.cc:492] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: No bootstrap required, opened a new log
I20260928 08:58:48.848279 2014 ts_tablet_manager.cc:1397] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:58:48.848368 2013 tablet_bootstrap.cc:492] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: No bootstrap required, opened a new log
I20260928 08:58:48.848435 2013 ts_tablet_manager.cc:1397] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:58:48.849728 2015 raft_consensus.cc:348] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.849946 2015 raft_consensus.cc:374] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.850000 2015 raft_consensus.cc:729] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479fc3259b584e84882ac78d0fe6f236, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.850145 2015 consensus_queue.cc:260] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.850399 2015 ts_tablet_manager.cc:1428] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:58:48.850488 1990 heartbeater.cc:503] Master 127.0.98.62:38157 was elected leader, sending a full tablet report...
I20260928 08:58:48.850628 2013 raft_consensus.cc:348] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.850775 2013 raft_consensus.cc:374] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.850816 2013 raft_consensus.cc:729] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231aff02cbce456ca83bfa43440b1a17, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.850944 2013 consensus_queue.cc:260] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.850855 2014 raft_consensus.cc:348] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.851044 2014 raft_consensus.cc:374] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.851089 2014 raft_consensus.cc:729] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7472cae13746a399036a92785d5a07, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.851202 2013 ts_tablet_manager.cc:1428] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:58:48.851202 2014 consensus_queue.cc:260] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.851322 1710 heartbeater.cc:503] Master 127.0.98.62:38157 was elected leader, sending a full tablet report...
I20260928 08:58:48.851495 2014 ts_tablet_manager.cc:1428] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 08:58:48.851635 1850 heartbeater.cc:503] Master 127.0.98.62:38157 was elected leader, sending a full tablet report...
I20260928 08:58:48.891625 2019 raft_consensus.cc:482] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:48.891808 2019 raft_consensus.cc:504] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.892190 2019 leader_election.cc:288] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541), 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049)
I20260928 08:58:48.895612 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.892394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35541 (local address 127.0.98.3:42311)
0928 08:58:48.892467 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.892479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.892524 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.892717 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.892718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.892720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:48.892794 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.892796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.893560 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.893561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.894536 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.894539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.895317 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.895318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.895397 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.895417 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.895473 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.895516 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 08:58:48.895625 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.892469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42311 (local address 127.0.98.2:35541)
0928 08:58:48.892521 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.892524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.892536 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:58:48.892592 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.892593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.892609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.892714 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:48.892887 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.893487 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.894614 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.895223 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.895300 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.895328 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.895497 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.895549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:58:48.895679 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.892514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57271 (local address 127.0.98.1:35049)
0928 08:58:48.892557 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.892558 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.892733 (+ 175us) server_negotiation.cc:407] Connection header received
0928 08:58:48.892815 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.892817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.892828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.892885 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:48.893047 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.893598 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.894745 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.895286 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.895354 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.895364 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.895564 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.895602 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:58:48.895741 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.892432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.98.3:57271)
0928 08:58:48.892678 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.892689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.892747 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.892901 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.892902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.892903 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:48.892976 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.892978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.893676 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.893677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.894645 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.894647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.895387 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.895389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.895464 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.895469 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.895552 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.895593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
W20260928 08:58:48.895875 1851 tablet.cc:2369] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:58:48.896068 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baefe10ee764b029bba89b00618519e" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231aff02cbce456ca83bfa43440b1a17" is_pre_election: true
I20260928 08:58:48.896243 1664 raft_consensus.cc:2454] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 0.
I20260928 08:58:48.896376 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baefe10ee764b029bba89b00618519e" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7472cae13746a399036a92785d5a07" is_pre_election: true
I20260928 08:58:48.896466 1877 leader_election.cc:302] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231aff02cbce456ca83bfa43440b1a17, 479fc3259b584e84882ac78d0fe6f236; no voters:
I20260928 08:58:48.896533 1804 raft_consensus.cc:2454] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 0.
I20260928 08:58:48.896600 2019 raft_consensus.cc:2781] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:48.896696 2019 raft_consensus.cc:482] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:48.896729 2019 raft_consensus.cc:3037] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.897722 2019 raft_consensus.cc:504] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.897873 2019 leader_election.cc:288] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 election: Requested vote from peers ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541), 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049)
I20260928 08:58:48.898002 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baefe10ee764b029bba89b00618519e" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7472cae13746a399036a92785d5a07"
I20260928 08:58:48.898029 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baefe10ee764b029bba89b00618519e" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231aff02cbce456ca83bfa43440b1a17"
I20260928 08:58:48.898085 1804 raft_consensus.cc:3037] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.898120 1664 raft_consensus.cc:3037] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.898885 1804 raft_consensus.cc:2454] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 1.
I20260928 08:58:48.898953 1664 raft_consensus.cc:2454] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 1.
I20260928 08:58:48.899132 1877 leader_election.cc:302] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231aff02cbce456ca83bfa43440b1a17, 479fc3259b584e84882ac78d0fe6f236; no voters:
I20260928 08:58:48.899231 2019 raft_consensus.cc:2781] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:48.899385 2019 raft_consensus.cc:686] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 1 LEADER]: Becoming Leader. State: Replica: 479fc3259b584e84882ac78d0fe6f236, State: Running, Role: LEADER
I20260928 08:58:48.899494 2019 consensus_queue.cc:237] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } }
I20260928 08:58:48.900347 1507 catalog_manager.cc:5495] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "479fc3259b584e84882ac78d0fe6f236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260928 08:58:48.915145 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.912547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.0.98.3:41847)
0928 08:58:48.912594 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.912596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.912607 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:58:48.912760 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.912761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.912782 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.912860 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:48.912995 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.913464 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.914286 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.914508 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.914578 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.914791 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:48.914899 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:48.914988 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.915036 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.915060 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:58:48.916909 1664 raft_consensus.cc:1261] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Refusing update from remote peer 479fc3259b584e84882ac78d0fe6f236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:48.916910 1804 raft_consensus.cc:1261] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Refusing update from remote peer 479fc3259b584e84882ac78d0fe6f236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:48.917219 2019 consensus_queue.cc:1035] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:48.917372 2023 consensus_queue.cc:1035] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:48.919842 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334239964840939520 to 7334239964773064704. Current Snapshot: MvccSnapshot[applied={T|T < 7334239964840939520}]
I20260928 08:58:48.919811 2026 mvcc.cc:205] Tried to move back new op lower bound from 7334239964840939520 to 7334239964773064704. Current Snapshot: MvccSnapshot[applied={T|T < 7334239964840939520}]
I20260928 08:58:48.924222 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:48.924321 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:48.925114 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: Sending DeleteTablet for 3 replicas of tablet 9baefe10ee764b029bba89b00618519e
I20260928 08:58:48.925482 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 9baefe10ee764b029bba89b00618519e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59392
I20260928 08:58:48.925480 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9baefe10ee764b029bba89b00618519e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42754
I20260928 08:58:48.925575 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 9baefe10ee764b029bba89b00618519e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39756
I20260928 08:58:48.925786 2037 tablet_replica.cc:330] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: stopping tablet replica
I20260928 08:58:48.925921 2037 raft_consensus.cc:2229] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:48.926017 2035 tablet_replica.cc:330] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: stopping tablet replica
I20260928 08:58:48.926113 2035 raft_consensus.cc:2229] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:58:48.926216 2037 raft_consensus.cc:2258] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:48.926210 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:48.926337 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:48.926697 2037 ts_tablet_manager.cc:1905] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:48.927623 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e04e92ac95d34bcc8b7c458cb18e3475 (DEFAULT_TABLE table=test-table [id=415be70eb923479d84fade8ef974215e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:48.927662 1784 tablet_service.cc:1467] Processing CreateTablet for tablet e04e92ac95d34bcc8b7c458cb18e3475 (DEFAULT_TABLE table=test-table [id=415be70eb923479d84fade8ef974215e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:48.927753 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04e92ac95d34bcc8b7c458cb18e3475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.927802 1644 tablet_service.cc:1467] Processing CreateTablet for tablet e04e92ac95d34bcc8b7c458cb18e3475 (DEFAULT_TABLE table=test-table [id=415be70eb923479d84fade8ef974215e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:48.927914 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04e92ac95d34bcc8b7c458cb18e3475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.928785 2015 tablet_bootstrap.cc:492] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Bootstrap starting.
I20260928 08:58:48.929364 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04e92ac95d34bcc8b7c458cb18e3475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.929558 2013 tablet_bootstrap.cc:492] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Bootstrap starting.
I20260928 08:58:48.929632 2015 tablet_bootstrap.cc:654] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.930176 2013 tablet_bootstrap.cc:654] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.930526 2014 tablet_bootstrap.cc:492] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Bootstrap starting.
I20260928 08:58:48.926647 2035 raft_consensus.cc:2258] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:48.930912 2014 tablet_bootstrap.cc:654] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.930958 2036 tablet_replica.cc:330] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: stopping tablet replica
I20260928 08:58:48.931056 2036 raft_consensus.cc:2229] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:48.931165 2035 ts_tablet_manager.cc:1905] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:48.931161 2036 raft_consensus.cc:2258] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:48.931268 2037 ts_tablet_manager.cc:1918] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:58:48.931349 2037 log.cc:1196] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/9baefe10ee764b029bba89b00618519e
I20260928 08:58:48.931610 2037 ts_tablet_manager.cc:1939] T 9baefe10ee764b029bba89b00618519e P ba7472cae13746a399036a92785d5a07: Deleting consensus metadata
I20260928 08:58:48.931685 2013 tablet_bootstrap.cc:492] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: No bootstrap required, opened a new log
I20260928 08:58:48.931739 2013 ts_tablet_manager.cc:1397] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:48.931915 2013 raft_consensus.cc:348] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.931993 2013 raft_consensus.cc:374] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.932040 2013 raft_consensus.cc:729] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231aff02cbce456ca83bfa43440b1a17, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.932096 2013 consensus_queue.cc:260] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.932221 2013 ts_tablet_manager.cc:1428] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:48.932242 2014 tablet_bootstrap.cc:492] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: No bootstrap required, opened a new log
I20260928 08:58:48.932286 2014 ts_tablet_manager.cc:1397] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:48.932360 1492 catalog_manager.cc:4841] TS ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541): tablet 9baefe10ee764b029bba89b00618519e (table test-table [id=b0701a6267d642d9859be720e9c39c8f]) successfully deleted
I20260928 08:58:48.932446 2014 raft_consensus.cc:348] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.932515 2014 raft_consensus.cc:374] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.932541 2014 raft_consensus.cc:729] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7472cae13746a399036a92785d5a07, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.932577 2014 consensus_queue.cc:260] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.932675 2014 ts_tablet_manager.cc:1428] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:48.932834 2035 ts_tablet_manager.cc:1918] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:58:48.932912 2035 log.cc:1196] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/9baefe10ee764b029bba89b00618519e
I20260928 08:58:48.932955 2036 ts_tablet_manager.cc:1905] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:48.933434 2035 ts_tablet_manager.cc:1939] T 9baefe10ee764b029bba89b00618519e P 479fc3259b584e84882ac78d0fe6f236: Deleting consensus metadata
I20260928 08:58:48.933822 1493 catalog_manager.cc:4841] TS 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3:41847): tablet 9baefe10ee764b029bba89b00618519e (table test-table [id=b0701a6267d642d9859be720e9c39c8f]) successfully deleted
I20260928 08:58:48.934477 2036 ts_tablet_manager.cc:1918] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:58:48.934643 2036 log.cc:1196] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/9baefe10ee764b029bba89b00618519e
I20260928 08:58:48.934952 2036 ts_tablet_manager.cc:1939] T 9baefe10ee764b029bba89b00618519e P 231aff02cbce456ca83bfa43440b1a17: Deleting consensus metadata
I20260928 08:58:48.935321 1493 catalog_manager.cc:4841] TS 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049): tablet 9baefe10ee764b029bba89b00618519e (table test-table [id=b0701a6267d642d9859be720e9c39c8f]) successfully deleted
I20260928 08:58:48.937052 2015 tablet_bootstrap.cc:492] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: No bootstrap required, opened a new log
I20260928 08:58:48.937109 2015 ts_tablet_manager.cc:1397] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 08:58:48.937290 2015 raft_consensus.cc:348] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.937379 2015 raft_consensus.cc:374] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.937402 2015 raft_consensus.cc:729] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479fc3259b584e84882ac78d0fe6f236, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.937440 2015 consensus_queue.cc:260] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.937574 2015 ts_tablet_manager.cc:1428] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.945719 2019 raft_consensus.cc:482] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:48.945806 2019 raft_consensus.cc:504] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.945959 2019 leader_election.cc:288] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049), ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541)
I20260928 08:58:48.946090 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e92ac95d34bcc8b7c458cb18e3475" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231aff02cbce456ca83bfa43440b1a17" is_pre_election: true
I20260928 08:58:48.946123 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e92ac95d34bcc8b7c458cb18e3475" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7472cae13746a399036a92785d5a07" is_pre_election: true
I20260928 08:58:48.946211 1804 raft_consensus.cc:2454] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 0.
I20260928 08:58:48.946218 1664 raft_consensus.cc:2454] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 0.
I20260928 08:58:48.946352 1876 leader_election.cc:302] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479fc3259b584e84882ac78d0fe6f236, ba7472cae13746a399036a92785d5a07; no voters:
I20260928 08:58:48.946439 2019 raft_consensus.cc:2781] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:48.946484 2019 raft_consensus.cc:482] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:48.946506 2019 raft_consensus.cc:3037] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.947314 2019 raft_consensus.cc:504] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.947438 2019 leader_election.cc:288] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 election: Requested vote from peers 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049), ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541)
I20260928 08:58:48.947584 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e92ac95d34bcc8b7c458cb18e3475" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7472cae13746a399036a92785d5a07"
I20260928 08:58:48.947652 1804 raft_consensus.cc:3037] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.947595 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04e92ac95d34bcc8b7c458cb18e3475" candidate_uuid: "479fc3259b584e84882ac78d0fe6f236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231aff02cbce456ca83bfa43440b1a17"
I20260928 08:58:48.947700 1664 raft_consensus.cc:3037] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:48.948360 1664 raft_consensus.cc:2454] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 1.
I20260928 08:58:48.948424 1804 raft_consensus.cc:2454] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479fc3259b584e84882ac78d0fe6f236 in term 1.
I20260928 08:58:48.948576 1876 leader_election.cc:302] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479fc3259b584e84882ac78d0fe6f236, ba7472cae13746a399036a92785d5a07; no voters:
I20260928 08:58:48.948756 2019 raft_consensus.cc:2781] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:48.948814 2019 raft_consensus.cc:686] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 1 LEADER]: Becoming Leader. State: Replica: 479fc3259b584e84882ac78d0fe6f236, State: Running, Role: LEADER
I20260928 08:58:48.948872 2019 consensus_queue.cc:237] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.949409 1506 catalog_manager.cc:5495] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "479fc3259b584e84882ac78d0fe6f236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } health_report { overall_health: HEALTHY } } }
Sep 28 08:58:48 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:48.985144 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.977429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.0.98.62:38157)
0928 08:58:48.977499 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:48.977502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:48.977645 (+ 143us) server_negotiation.cc:407] Connection header received
0928 08:58:48.977679 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:48.977680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:48.977691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:48.977755 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:48.978334 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.978933 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.979998 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.980180 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.980240 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.982626 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:48.982627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:48.982627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:48.982633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:48.984001 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.984368 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.984369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.984369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.984417 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:48.984598 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:48.984599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:48.984600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:48.984740 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:48.984812 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:48.984956 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:58:48.985037 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:58:48.985230 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:48.977381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.0.1:46816)
0928 08:58:48.977545 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:48.977556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:48.977629 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:48.977771 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:48.977772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:48.978190 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:48.978246 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.978249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.979013 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.979014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.979899 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:48.979901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:48.980255 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:48.980256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:48.980316 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:48.980967 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:48.980972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:48.982484 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:48.984085 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.984092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.984094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.984288 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.984477 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:48.984478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:48.984479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:48.984533 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:48.984802 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:48.984809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:48.984854 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:58:48.984934 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:58:48.985110 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":71,"thread_start_us":98,"threads_started":1}
I20260928 08:58:48.987171 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46816:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:58:48.987385 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:48.988799 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 6db7513d1b0f46d79722e17c64d3b9da (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d77bb903c6e04747a27b73f575811a78]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:58:48.988808 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6db7513d1b0f46d79722e17c64d3b9da (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d77bb903c6e04747a27b73f575811a78]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:58:48.988930 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db7513d1b0f46d79722e17c64d3b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.988946 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db7513d1b0f46d79722e17c64d3b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.989988 2014 tablet_bootstrap.cc:492] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: Bootstrap starting.
I20260928 08:58:48.990100 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 6db7513d1b0f46d79722e17c64d3b9da (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d77bb903c6e04747a27b73f575811a78]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:58:48.990247 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db7513d1b0f46d79722e17c64d3b9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:48.990427 2014 tablet_bootstrap.cc:654] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.990823 2015 tablet_bootstrap.cc:492] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: Bootstrap starting.
I20260928 08:58:48.991127 2015 tablet_bootstrap.cc:654] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.991529 2014 tablet_bootstrap.cc:492] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: No bootstrap required, opened a new log
I20260928 08:58:48.991581 2014 ts_tablet_manager.cc:1397] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:58:48.991628 2015 tablet_bootstrap.cc:492] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: No bootstrap required, opened a new log
I20260928 08:58:48.991667 2015 ts_tablet_manager.cc:1397] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:48.991724 2014 raft_consensus.cc:348] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.991808 2014 raft_consensus.cc:374] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.991765 2015 raft_consensus.cc:348] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.991832 2014 raft_consensus.cc:729] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7472cae13746a399036a92785d5a07, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.991837 2015 raft_consensus.cc:374] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.991858 2015 raft_consensus.cc:729] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479fc3259b584e84882ac78d0fe6f236, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.991851 2013 tablet_bootstrap.cc:492] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: Bootstrap starting.
I20260928 08:58:48.991876 2014 consensus_queue.cc:260] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.991899 2015 consensus_queue.cc:260] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.991964 2014 ts_tablet_manager.cc:1428] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.991986 2015 ts_tablet_manager.cc:1428] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:48.992108 2023 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:48.992192 2023 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:48.992246 2013 tablet_bootstrap.cc:654] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:48.992724 2021 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:48.992836 2021 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 08:58:48.992997 1711 tablet.cc:2369] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:58:48.993574 2013 tablet_bootstrap.cc:492] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: No bootstrap required, opened a new log
I20260928 08:58:48.993618 2013 ts_tablet_manager.cc:1397] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:48.993764 2013 raft_consensus.cc:348] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.993829 2013 raft_consensus.cc:374] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:48.993844 2013 raft_consensus.cc:729] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231aff02cbce456ca83bfa43440b1a17, State: Initialized, Role: FOLLOWER
I20260928 08:58:48.993892 2013 consensus_queue.cc:260] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:48.993983 2013 ts_tablet_manager.cc:1428] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:48.994035 2020 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:48.994103 2020 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:49.002383 2021 raft_consensus.cc:482] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:49.002455 2021 raft_consensus.cc:504] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:49.002787 2021 leader_election.cc:288] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049), 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3:41847)
I20260928 08:58:49.006465 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.003103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53887 (local address 127.0.98.1:35049)
0928 08:58:49.003172 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.003173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.003181 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:58:49.003215 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.003216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.003227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.003287 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:49.003512 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.004049 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.005509 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.006106 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.006185 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.006206 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.006361 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.006403 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 08:58:49.006474 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.002922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35049 (local address 127.0.98.2:53887)
0928 08:58:49.002964 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:49.002975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:49.003015 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:49.003336 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:49.003337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:49.003338 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:49.003413 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.003415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.004126 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.004128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.005413 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.005416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.006208 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.006209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.006285 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.006289 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:58:49.006344 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:58:49.006385 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":1}
I20260928 08:58:49.006700 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db7513d1b0f46d79722e17c64d3b9da" candidate_uuid: "ba7472cae13746a399036a92785d5a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231aff02cbce456ca83bfa43440b1a17" is_pre_election: true
I20260928 08:58:49.006754 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.003094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33091 (local address 127.0.98.3:41847)
0928 08:58:49.003134 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.003136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.003245 (+ 109us) server_negotiation.cc:407] Connection header received
0928 08:58:49.003307 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.003308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.003317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.003367 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:49.003560 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.004295 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.005893 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.006387 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.006425 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.006433 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.006657 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.006697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:58:49.006779 1664 raft_consensus.cc:2454] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba7472cae13746a399036a92785d5a07 in term 0.
I20260928 08:58:49.006943 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.003019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41847 (local address 127.0.98.2:33091)
0928 08:58:49.003184 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:49.003192 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:49.003256 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:49.003377 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:49.003379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:49.003381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:49.003467 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.003469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.004385 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.004385 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.005796 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.005799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.006487 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.006488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.006570 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.006573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:58:49.006812 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 08:58:49.006869 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260928 08:58:49.006934 1737 leader_election.cc:302] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231aff02cbce456ca83bfa43440b1a17, ba7472cae13746a399036a92785d5a07; no voters:
I20260928 08:58:49.007051 2021 raft_consensus.cc:2781] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:49.007086 2021 raft_consensus.cc:482] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:49.007112 2021 raft_consensus.cc:3037] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:49.007272 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db7513d1b0f46d79722e17c64d3b9da" candidate_uuid: "ba7472cae13746a399036a92785d5a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479fc3259b584e84882ac78d0fe6f236" is_pre_election: true
I20260928 08:58:49.007388 1944 raft_consensus.cc:2454] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba7472cae13746a399036a92785d5a07 in term 0.
I20260928 08:58:49.007892 2021 raft_consensus.cc:504] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:49.008018 2021 leader_election.cc:288] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [CANDIDATE]: Term 1 election: Requested vote from peers 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049), 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3:41847)
I20260928 08:58:49.008193 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db7513d1b0f46d79722e17c64d3b9da" candidate_uuid: "ba7472cae13746a399036a92785d5a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479fc3259b584e84882ac78d0fe6f236"
I20260928 08:58:49.008190 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db7513d1b0f46d79722e17c64d3b9da" candidate_uuid: "ba7472cae13746a399036a92785d5a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "231aff02cbce456ca83bfa43440b1a17"
I20260928 08:58:49.008270 1664 raft_consensus.cc:3037] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:49.008270 1944 raft_consensus.cc:3037] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:49.008903 1664 raft_consensus.cc:2454] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba7472cae13746a399036a92785d5a07 in term 1.
I20260928 08:58:49.009141 1737 leader_election.cc:302] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 231aff02cbce456ca83bfa43440b1a17, ba7472cae13746a399036a92785d5a07; no voters:
I20260928 08:58:49.009172 1944 raft_consensus.cc:2454] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba7472cae13746a399036a92785d5a07 in term 1.
I20260928 08:58:49.009222 2021 raft_consensus.cc:2781] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:49.009351 2021 raft_consensus.cc:686] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [term 1 LEADER]: Becoming Leader. State: Replica: ba7472cae13746a399036a92785d5a07, State: Running, Role: LEADER
I20260928 08:58:49.009428 2021 consensus_queue.cc:237] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } }
I20260928 08:58:49.009757 2052 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: TxnStatusTablet state changed. Reason: New leader ba7472cae13746a399036a92785d5a07. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.009830 2052 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:58:49.010052 1506 catalog_manager.cc:5495] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba7472cae13746a399036a92785d5a07 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } health_report { overall_health: UNKNOWN } } }
I20260928 08:58:49.010416 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:58:49.010497 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:58:49.019682 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.016516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.0.98.2:35541)
0928 08:58:49.016552 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.016566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.016575 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:49.016624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.016625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.016666 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.016725 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:49.016865 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.017608 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.018685 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.019025 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.019100 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.019311 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:49.019426 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:49.019504 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.019590 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.019610 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:58:49.019701 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.016399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35541 (local address 127.0.0.1:39760)
0928 08:58:49.016477 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:49.016490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:49.016542 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:49.016741 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:49.016742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:49.016743 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:49.016790 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.016793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.017702 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.017703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.018587 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.018591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.019117 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.019118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.019182 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.019207 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:49.019511 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:49.019520 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 08:58:49.019572 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:58:49.019625 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 08:58:49.020825 1664 raft_consensus.cc:1261] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Refusing update from remote peer ba7472cae13746a399036a92785d5a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:49.020892 1944 raft_consensus.cc:1261] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Refusing update from remote peer ba7472cae13746a399036a92785d5a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:49.021230 2021 consensus_queue.cc:1035] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:49.021348 2052 consensus_queue.cc:1035] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:49.021833 2020 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: TxnStatusTablet state changed. Reason: New leader ba7472cae13746a399036a92785d5a07. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.021914 2020 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:49.022089 2023 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: TxnStatusTablet state changed. Reason: New leader ba7472cae13746a399036a92785d5a07. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.022210 2023 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:49.022723 2021 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.022797 2021 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:58:49.023201 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334239965267021824 to 7334239965222952960. Current Snapshot: MvccSnapshot[applied={T|T < 7334239965267021824}]
I20260928 08:58:49.023267 2020 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.023340 2020 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P 231aff02cbce456ca83bfa43440b1a17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:49.025399 2052 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.025480 2052 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P ba7472cae13746a399036a92785d5a07: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:58:49.026537 2023 tablet_replica.cc:439] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba7472cae13746a399036a92785d5a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 } } peers { permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 } } peers { permanent_uuid: "479fc3259b584e84882ac78d0fe6f236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41847 } } }
I20260928 08:58:49.026620 2023 tablet_replica.cc:442] T 6db7513d1b0f46d79722e17c64d3b9da P 479fc3259b584e84882ac78d0fe6f236: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:49.029616 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.026607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35541 (local address 127.0.98.3:47579)
0928 08:58:49.026688 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:49.026699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:49.026741 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:49.026862 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:49.026863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:49.026864 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:49.026923 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.026925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.027795 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.027796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.028738 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.028741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.029207 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.029208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.029261 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.029275 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:49.029453 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:49.029454 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:58:49.029492 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 08:58:49.029529 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 08:58:49.029915 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.026700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47579 (local address 127.0.98.2:35541)
0928 08:58:49.026730 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.026733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.026740 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:58:49.026799 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.026800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.026810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.026858 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:49.026992 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.027704 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.028828 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.029122 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.029177 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.029362 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:49.029393 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:49.029434 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.029836 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.029853 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:58:49.032256 1664 raft_consensus.cc:1261] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Refusing update from remote peer 479fc3259b584e84882ac78d0fe6f236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:49.032336 1804 raft_consensus.cc:1261] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Refusing update from remote peer 479fc3259b584e84882ac78d0fe6f236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:49.032534 2023 consensus_queue.cc:1035] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "231aff02cbce456ca83bfa43440b1a17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:49.032661 2019 consensus_queue.cc:1035] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba7472cae13746a399036a92785d5a07" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:49.041961 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.038503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41847 (local address 127.0.98.2:33405)
0928 08:58:49.038554 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:49.038566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:49.038610 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:49.038787 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:49.038788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:49.038790 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:49.038848 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.038849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.039752 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.039753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.040668 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:49.040671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.041525 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.041526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:49.041592 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.041608 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:49.041802 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:49.041804 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:58:49.041843 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 08:58:49.041884 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 08:58:49.042387 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.038543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33405 (local address 127.0.98.3:41847)
0928 08:58:49.038593 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.038595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.038604 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:49.038666 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.038667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.038680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.038729 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:49.038923 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.039657 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.040968 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.041407 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.041486 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.041698 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:49.041732 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:49.041774 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.042287 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.042314 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 08:58:49.049573 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:49.049680 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:49.050254 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 963a62cdbe8646b180035e27fa172721: Sending DeleteTablet for 3 replicas of tablet e04e92ac95d34bcc8b7c458cb18e3475
I20260928 08:58:49.050486 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet e04e92ac95d34bcc8b7c458cb18e3475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59392
I20260928 08:58:49.050534 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet e04e92ac95d34bcc8b7c458cb18e3475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39756
I20260928 08:58:49.050619 2036 tablet_replica.cc:330] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: stopping tablet replica
I20260928 08:58:49.050678 2036 raft_consensus.cc:2229] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:49.050716 2036 raft_consensus.cc:2258] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:49.050786 2037 tablet_replica.cc:330] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: stopping tablet replica
I20260928 08:58:49.050832 2037 raft_consensus.cc:2229] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:49.050868 2037 raft_consensus.cc:2258] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:49.051098 2036 ts_tablet_manager.cc:1905] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:49.051215 2037 ts_tablet_manager.cc:1905] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:49.052852 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet e04e92ac95d34bcc8b7c458cb18e3475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42754
I20260928 08:58:49.052942 2035 tablet_replica.cc:330] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: stopping tablet replica
I20260928 08:58:49.052990 2035 raft_consensus.cc:2229] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:58:49.052995 2037 ts_tablet_manager.cc:1918] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:49.053045 2035 raft_consensus.cc:2258] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:49.053053 2037 log.cc:1196] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/e04e92ac95d34bcc8b7c458cb18e3475
I20260928 08:58:49.053289 2037 ts_tablet_manager.cc:1939] T e04e92ac95d34bcc8b7c458cb18e3475 P ba7472cae13746a399036a92785d5a07: Deleting consensus metadata
I20260928 08:58:49.053367 2035 ts_tablet_manager.cc:1905] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:49.053543 1492 catalog_manager.cc:4841] TS ba7472cae13746a399036a92785d5a07 (127.0.98.2:35541): tablet e04e92ac95d34bcc8b7c458cb18e3475 (table test-table [id=415be70eb923479d84fade8ef974215e]) successfully deleted
I20260928 08:58:49.053592 2036 ts_tablet_manager.cc:1918] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:49.053637 2036 log.cc:1196] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/e04e92ac95d34bcc8b7c458cb18e3475
I20260928 08:58:49.053833 2036 ts_tablet_manager.cc:1939] T e04e92ac95d34bcc8b7c458cb18e3475 P 231aff02cbce456ca83bfa43440b1a17: Deleting consensus metadata
I20260928 08:58:49.054119 1493 catalog_manager.cc:4841] TS 231aff02cbce456ca83bfa43440b1a17 (127.0.98.1:35049): tablet e04e92ac95d34bcc8b7c458cb18e3475 (table test-table [id=415be70eb923479d84fade8ef974215e]) successfully deleted
I20260928 08:58:49.054764 2035 ts_tablet_manager.cc:1918] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:49.054834 2035 log.cc:1196] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/e04e92ac95d34bcc8b7c458cb18e3475
I20260928 08:58:49.055051 2035 ts_tablet_manager.cc:1939] T e04e92ac95d34bcc8b7c458cb18e3475 P 479fc3259b584e84882ac78d0fe6f236: Deleting consensus metadata
I20260928 08:58:49.055411 1493 catalog_manager.cc:4841] TS 479fc3259b584e84882ac78d0fe6f236 (127.0.98.3:41847): tablet e04e92ac95d34bcc8b7c458cb18e3475 (table test-table [id=415be70eb923479d84fade8ef974215e]) successfully deleted
Sep 28 08:58:49 dist-test-slave-286n krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:58:49.074039 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.064940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.1:35049)
0928 08:58:49.064999 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.065001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.065093 (+ 92us) server_negotiation.cc:407] Connection header received
0928 08:58:49.065167 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.065168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.065182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.065277 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:49.065991 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.066467 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.067494 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.067678 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.067730 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.071042 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:49.071053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:49.071054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:49.071061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:49.072916 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:49.073229 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:49.073233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:49.073234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:49.073301 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:49.073468 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:49.073469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:49.073469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:49.073667 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:49.073811 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.073849 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.073928 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 08:58:49.074368 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59400
I20260928 08:58:49.081517 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:49.075343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.0.98.62:38157)
0928 08:58:49.075389 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:49.075391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:49.075416 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:58:49.075503 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:49.075504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:49.075515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:49.075593 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:49.076047 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.076848 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.077495 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:49.077801 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:49.077867 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:49.078927 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:49.078929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:49.078929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:49.078936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:49.080398 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:49.080752 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:49.080753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:49.080754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:49.080822 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:49.080993 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:49.080994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:49.080994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:49.081135 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:49.081235 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:49.081325 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 08:58:49.081412 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 08:58:49.081770 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46822
I20260928 08:58:49.082473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1579
I20260928 08:58:49.088999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1719
I20260928 08:58:49.093395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1859
I20260928 08:58:49.098769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 1475
2026-09-28T08:58:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2138 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:49 dist-test-slave-286n krb5kdc[2073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:49 dist-test-slave-286n krb5kdc[2073](info): set up 2 sockets
Sep 28 08:58:49 dist-test-slave-286n krb5kdc[2073](info): commencing operation
krb5kdc: starting...
W20260928 08:58:50.152285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:58:50.279462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:50.358815 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:50.359059 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:50.359094 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:50.360220 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:50.360282 2086 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:58:50.360303 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:50.360323 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:50.360349 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:50.360370 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:50.361831 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45427
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:50.362102 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:50.362313 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:50.367262 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:50.367503 2086 server_base.cc:1034] running on GCE node
W20260928 08:58:50.367302 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:50.367273 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:50.367933 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:50.368208 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:50.369362 2086 hybrid_clock.cc:648] HybridClock initialized: now 1790585930369356 us; error 43 us; skew 500 ppm
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:50.371831 2086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:50.372246 2086 webserver.cc:460] Webserver started at http://127.0.98.62:43313/ using document root <none> and password file <none>
I20260928 08:58:50.372454 2086 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:50.372517 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:50.372695 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:50.373709 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "3b98194b8cc34de891fd7d563a66ec62"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "e511a62f56b5d671e3975954b5ea5729"
server_key_iv: "bf782f278a2e481cab7b4e4b70353b4e"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.374069 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b98194b8cc34de891fd7d563a66ec62"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "e511a62f56b5d671e3975954b5ea5729"
server_key_iv: "bf782f278a2e481cab7b4e4b70353b4e"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.375411 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:50.376133 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.376305 2086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:50.376402 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "3b98194b8cc34de891fd7d563a66ec62"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "e511a62f56b5d671e3975954b5ea5729"
server_key_iv: "bf782f278a2e481cab7b4e4b70353b4e"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.376480 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:50.392189 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:50.394287 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:50.394464 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:50.399219 2086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45427
I20260928 08:58:50.399267 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45427 every 8 connection(s)
I20260928 08:58:50.399585 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:50.400238 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:50.402751 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Bootstrap starting.
I20260928 08:58:50.403363 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:50.403669 2164 log.cc:824] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:50.404158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 2086
I20260928 08:58:50.404266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:50.404431 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: No bootstrap required, opened a new log
I20260928 08:58:50.404443 392 external_mini_cluster.cc:1428] Setting key cf3b8c057c9ffc5bc9bd737e9fc07d03
I20260928 08:58:50.406797 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b98194b8cc34de891fd7d563a66ec62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 08:58:50.406956 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:50.406999 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b98194b8cc34de891fd7d563a66ec62, State: Initialized, Role: FOLLOWER
I20260928 08:58:50.407140 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b98194b8cc34de891fd7d563a66ec62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 08:58:50.407228 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:50.407271 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:50.407320 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:50.407827 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b98194b8cc34de891fd7d563a66ec62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 08:58:50.407938 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b98194b8cc34de891fd7d563a66ec62; no voters:
I20260928 08:58:50.408111 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:50.408159 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:50.408293 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [term 1 LEADER]: Becoming Leader. State: Replica: 3b98194b8cc34de891fd7d563a66ec62, State: Running, Role: LEADER
I20260928 08:58:50.408368 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:50.408384 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b98194b8cc34de891fd7d563a66ec62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 08:58:50.409076 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b98194b8cc34de891fd7d563a66ec62. Latest consensus state: current_term: 1 leader_uuid: "3b98194b8cc34de891fd7d563a66ec62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b98194b8cc34de891fd7d563a66ec62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260928 08:58:50.409199 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:50.409411 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:50.409075 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b98194b8cc34de891fd7d563a66ec62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b98194b8cc34de891fd7d563a66ec62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260928 08:58:50.409492 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:50.409938 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:50.411523 2178 catalog_manager.cc:1347] Generated new cluster ID: 101b594c56bf4f12b9a4b75259127547
I20260928 08:58:50.411574 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:50.417724 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.404981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.0.98.62:45427)
0928 08:58:50.405192 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.405200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.405226 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:58:50.406790 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.406799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.406853 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.407016 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.407715 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.409110 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.410165 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.410729 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.410850 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.413552 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.413556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.413556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.413568 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.415504 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.415885 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.415888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.415889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.416556 (+ 667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.416781 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.416782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.416782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.417114 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.417294 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.417363 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.417504 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260928 08:58:50.421263 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:50.421947 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:58:50.428520 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Generated new TSK 0
I20260928 08:58:50.428736 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:58:50.455219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:50.535987 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:50.536196 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:50.536222 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:50.537437 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:50.537482 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:50.537503 2190 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:58:50.537551 2190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:58:50.537580 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:58:50.539059 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:50.539286 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:50.539496 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:50.544358 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:50.544358 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:50.544548 2190 server_base.cc:1034] running on GCE node
W20260928 08:58:50.544382 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:50.544871 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:50.545109 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:50.546258 2190 hybrid_clock.cc:648] HybridClock initialized: now 1790585930546248 us; error 37 us; skew 500 ppm
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:50.548671 2190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:58:50.549083 2190 webserver.cc:460] Webserver started at http://127.0.98.1:37035/ using document root <none> and password file <none>
I20260928 08:58:50.549299 2190 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:50.549363 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:50.549549 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:50.550572 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "6415149702a74917bbfc9eeb286128b3"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "2e25a500c7edda4c32e3602355245052"
server_key_iv: "1eac1555c1ef03ed349e3f51167be1d2"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.550926 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6415149702a74917bbfc9eeb286128b3"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "2e25a500c7edda4c32e3602355245052"
server_key_iv: "1eac1555c1ef03ed349e3f51167be1d2"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.552211 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:50.552968 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.553124 2190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.553216 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "6415149702a74917bbfc9eeb286128b3"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "2e25a500c7edda4c32e3602355245052"
server_key_iv: "1eac1555c1ef03ed349e3f51167be1d2"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.553290 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:50.572031 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:50.574003 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:50.574163 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:50.574673 2190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:50.574738 2190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.574779 2190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:50.574805 2190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.581571 2190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41429
I20260928 08:58:50.581677 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41429 every 8 connection(s)
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:50.581981 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:58:50.584960 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.575166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40969 (local address 127.0.98.62:45427)
0928 08:58:50.575241 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.575244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.575436 (+ 192us) server_negotiation.cc:407] Connection header received
0928 08:58:50.575902 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.575904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.575917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.576004 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.576864 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.577665 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.579224 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.579646 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.579752 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.582540 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.582542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.582542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.582552 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.583715 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.584093 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.584094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.584094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.584146 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.584346 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.584346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.584347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.584465 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.584516 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.584815 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.584898 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:58:50.585130 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.575125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:40969)
0928 08:58:50.575288 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.575328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.575804 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.576082 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.576086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.576447 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.576769 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.576774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.577753 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.577754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.579135 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.579138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.579775 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.579776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.579920 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.580448 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:50.580459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:50.582442 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:50.583802 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.583808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.583810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.584024 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.584187 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.584188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.584188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.584296 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.584520 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:50.584525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:50.584684 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.584793 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.584912 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260928 08:58:50.585606 2321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260928 08:58:50.585685 2321 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:50.585873 2321 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260928 08:58:50.586325 2117 ts_manager.cc:194] Registered new tserver with Master: 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429)
I20260928 08:58:50.587009 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40969
I20260928 08:58:50.590021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 2190
I20260928 08:58:50.590132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:58:50.590341 392 external_mini_cluster.cc:1428] Setting key 040f8f2aedc7f06618c94a097f0e7a78
I20260928 08:58:50.591948 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.586686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:34997)
0928 08:58:50.586849 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.586863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.586911 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.587021 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.587022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.587214 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.587279 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.587281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.587878 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.587879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.588340 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.588341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.588664 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.588665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.588738 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.589178 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:50.589187 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:50.589466 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:50.590985 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.590986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.590986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.591178 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.591385 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.591385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.591385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.591434 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.591706 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:50.591707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:50.591739 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.591800 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.591883 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260928 08:58:50.592067 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.586747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34997 (local address 127.0.98.62:45427)
0928 08:58:50.586794 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.586796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.586917 (+ 121us) server_negotiation.cc:407] Connection header received
0928 08:58:50.586944 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.586945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.586954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.587021 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.587341 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.587818 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.588407 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.588575 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.588624 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.589532 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.589532 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.589533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.589537 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.590887 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.591246 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.591247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.591248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.591322 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.591489 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.591490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.591490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.591627 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.591712 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.591849 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.591934 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:58:50.616029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:50.696756 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:50.696941 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:50.696966 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:50.698019 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:50.698055 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:50.698073 2330 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:58:50.698086 2330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:58:50.698102 2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:58:50.699426 2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:50.699591 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:50.699750 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:50.704571 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:50.704566 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:50.704608 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:50.704928 2330 server_base.cc:1034] running on GCE node
I20260928 08:58:50.705112 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:50.705340 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:50.706476 2330 hybrid_clock.cc:648] HybridClock initialized: now 1790585930706459 us; error 34 us; skew 500 ppm
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:50.709096 2330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:58:50.709605 2330 webserver.cc:460] Webserver started at http://127.0.98.2:39593/ using document root <none> and password file <none>
I20260928 08:58:50.709837 2330 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:50.709882 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:50.710019 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:50.711079 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "90f6550b2451441e9d70cbb6555babb7"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "07c1b3d23db1e07b5b2c702354328be2"
server_key_iv: "b7aa95f01b49401f67f119cbbc1448e4"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.711433 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90f6550b2451441e9d70cbb6555babb7"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "07c1b3d23db1e07b5b2c702354328be2"
server_key_iv: "b7aa95f01b49401f67f119cbbc1448e4"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.713043 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:50.713968 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.714264 2330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:50.714367 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "90f6550b2451441e9d70cbb6555babb7"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "07c1b3d23db1e07b5b2c702354328be2"
server_key_iv: "b7aa95f01b49401f67f119cbbc1448e4"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.714459 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:50.725925 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:50.728325 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:50.728448 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:50.729015 2330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:50.729059 2330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.729085 2330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:50.729139 2330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:50.736575 2330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43643
I20260928 08:58:50.736685 2460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43643 every 8 connection(s)
I20260928 08:58:50.736999 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:58:50.739302 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.729494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39661 (local address 127.0.98.62:45427)
0928 08:58:50.729564 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.729568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.729772 (+ 204us) server_negotiation.cc:407] Connection header received
0928 08:58:50.730289 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.730290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.730302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.730358 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.731133 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.731940 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.733608 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.733925 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.734002 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.736672 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.736673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.736674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.736682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.738170 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.738577 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.738577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.738578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.738627 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.738801 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.738802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.738802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.738908 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.738956 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.739180 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.739240 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:58:50.739569 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.729474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:39661)
0928 08:58:50.729658 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.729697 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.730190 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.730444 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.730448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.730766 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.731048 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.731053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.732048 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.732050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.733501 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.733504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.734037 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.734038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.734203 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.734745 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:50.734758 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:50.736519 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:50.738260 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.738265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.738267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.738508 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.738669 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.738669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.738669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.738753 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.738961 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:50.738965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:50.739076 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.739158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.739298 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260928 08:58:50.740049 2461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260928 08:58:50.740136 2461 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:50.740356 2461 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260928 08:58:50.740352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 2330
I20260928 08:58:50.740447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:58:50.740665 392 external_mini_cluster.cc:1428] Setting key 2deb99f8179bca5171065a097e18a1c8
I20260928 08:58:50.740758 2118 ts_manager.cc:194] Registered new tserver with Master: 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643)
I20260928 08:58:50.741261 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39661
I20260928 08:58:50.747625 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.741110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:56785)
0928 08:58:50.741235 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.741250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.741296 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.741846 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.741847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.742042 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.742115 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.742117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.743028 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.743029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.743517 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.743518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.743979 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.743980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.744042 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.744479 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:50.744502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:50.744878 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:50.746653 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.746653 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.746654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.746823 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.747027 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.747027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.747028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.747076 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.747400 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:50.747401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:50.747433 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.747489 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.747559 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260928 08:58:50.747699 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.741167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56785 (local address 127.0.98.62:45427)
0928 08:58:50.741723 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.741724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.741731 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:58:50.741746 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.741746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.741771 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.741828 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.742198 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.742933 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.743587 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.743892 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.743968 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.744961 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.744962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.744963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.744970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.746573 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.746892 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.746893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.746893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.746956 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.747130 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.747131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.747131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.747329 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.747409 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.747516 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.747595 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:58:50.765718 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:50.845568 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:50.845774 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:50.845799 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:50.846952 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:50.846994 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:50.847015 2470 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:58:50.847061 2470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:58:50.847087 2470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:58:50.848526 2470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:50.848802 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:50.849063 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:58:50.854102 2470 server_base.cc:1034] running on GCE node
W20260928 08:58:50.854079 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:50.854063 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:50.854069 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:50.854475 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:50.854724 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:50.855872 2470 hybrid_clock.cc:648] HybridClock initialized: now 1790585930855863 us; error 31 us; skew 500 ppm
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:50.858348 2470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:58:50.858796 2470 webserver.cc:460] Webserver started at http://127.0.98.3:39139/ using document root <none> and password file <none>
I20260928 08:58:50.858975 2470 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:50.859014 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:50.859102 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:50.860090 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "084d19799cdf4f90b43f0553751bfcf1"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "373651581db4678cd1e1f9833156cfe0"
server_key_iv: "14ec01d0a99dbb674ca24132a2ee9766"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.860383 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "084d19799cdf4f90b43f0553751bfcf1"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "373651581db4678cd1e1f9833156cfe0"
server_key_iv: "14ec01d0a99dbb674ca24132a2ee9766"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.861701 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:50.862331 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.862476 2470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:50.862560 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "084d19799cdf4f90b43f0553751bfcf1"
format_stamp: "Formatted at 2026-09-28 08:58:50 on dist-test-slave-286n"
server_key: "373651581db4678cd1e1f9833156cfe0"
server_key_iv: "14ec01d0a99dbb674ca24132a2ee9766"
server_key_version: "encryptionkey@0"
I20260928 08:58:50.862612 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:50.871619 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:50.873503 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:50.873610 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:50.874037 2470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:50.874085 2470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.874127 2470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:50.874183 2470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:50.880887 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43549
I20260928 08:58:50.880931 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43549 every 8 connection(s)
I20260928 08:58:50.881237 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:50.884275 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.874575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38677 (local address 127.0.98.62:45427)
0928 08:58:50.874642 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.874644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.874852 (+ 208us) server_negotiation.cc:407] Connection header received
0928 08:58:50.875385 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.875387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.875402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.875488 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.876302 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.877180 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.878754 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.879052 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.879108 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.881821 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.881823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.881824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.881832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.882972 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.883359 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.883360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.883361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.883421 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.883657 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.883658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.883658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.883773 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.883846 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.884103 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.884184 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:58:50.884470 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.874563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:38677)
0928 08:58:50.874744 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.874782 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.875259 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.875565 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.875569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.875921 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.876190 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.876196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.877281 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.877282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.878646 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.878648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.879133 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.879135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.879317 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.879946 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:50.879969 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:50.881729 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:50.883051 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.883057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.883059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.883273 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.883484 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.883485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.883485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.883593 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.883835 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:50.883841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:50.883977 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.884080 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.884238 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":120,"threads_started":1}
I20260928 08:58:50.884923 2601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260928 08:58:50.885000 2601 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:50.885185 2601 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260928 08:58:50.885569 2117 ts_manager.cc:194] Registered new tserver with Master: 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:50.886097 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38677
I20260928 08:58:50.890043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 2470
I20260928 08:58:50.890157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:58:50.890345 392 external_mini_cluster.cc:1428] Setting key 1d1c7b72379e4da6fbcbd3a91b7ce5ca
I20260928 08:58:50.891147 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.885994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40503 (local address 127.0.98.62:45427)
0928 08:58:50.886032 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.886034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.886145 (+ 111us) server_negotiation.cc:407] Connection header received
0928 08:58:50.886212 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.886213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.886222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.886261 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.886607 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.887091 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.887777 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.887999 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.888078 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.888895 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.888896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.888896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.888902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.889981 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.890360 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.890361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.890362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.890415 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.890592 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.890593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.890593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.890749 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.890828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.890976 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.891050 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 08:58:50.891171 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.885941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:40503)
0928 08:58:50.886063 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.886074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.886158 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.886289 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.886290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.886456 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.886525 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.886526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.887175 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.887176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.887685 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.887687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.888081 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.888081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.888160 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.888499 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:50.888503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:50.888808 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:50.890060 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.890061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.890061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.890283 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.890465 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:50.890466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:50.890467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:50.890520 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:50.890824 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:50.890825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:50.890870 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.890952 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.891043 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260928 08:58:50.891408 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:50.913650 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.906428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.98.62:45427)
0928 08:58:50.906476 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.906478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.906489 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:58:50.906569 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.906570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.906579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.906635 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.907206 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.907713 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.908467 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.908716 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.908785 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.911223 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:50.911224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:50.911224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:50.911232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:50.912517 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.912895 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.912897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.912897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.912959 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:50.913144 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:50.913145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:50.913146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:50.913256 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:50.913359 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.913466 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.913549 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:58:50.915136 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:50.916741 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:50.923442 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.919064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.0.98.1:41429)
0928 08:58:50.919943 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.919947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.919956 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:50.919990 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.919992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.920006 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.920090 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:50.920177 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.920939 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.922428 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.923051 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.923139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.923200 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.923309 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.923346 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":64,"threads_started":1}
I20260928 08:58:50.923435 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.918909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41429 (local address 127.0.0.1:34512)
0928 08:58:50.919207 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.919235 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.919291 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.920085 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.920086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.920088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:50.920122 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.920124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.921011 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.921012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.922315 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.922318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.923139 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.923140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.923215 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.923229 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.923296 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.923354 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":77,"thread_start_us":52,"threads_started":1}
I20260928 08:58:50.923633 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.919002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.3:43549)
0928 08:58:50.919670 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.919675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.919684 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:50.919721 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.919723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.919736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.919817 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:50.919957 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.921055 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.922396 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.923120 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.923187 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.923240 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.923482 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.923523 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":89,"threads_started":1}
I20260928 08:58:50.923651 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.918907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43549 (local address 127.0.0.1:57536)
0928 08:58:50.919186 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.919224 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.919291 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.919816 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.919817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.919819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:50.919876 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.919878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.921125 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.921126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.922304 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.922306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.923275 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.923276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.923338 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.923343 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.923396 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.923439 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":45,"thread_start_us":53,"threads_started":1}
I20260928 08:58:50.923952 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.919194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.0.98.2:43643)
0928 08:58:50.919378 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.919382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.919394 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:58:50.919439 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.919440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.919452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.919539 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:50.919702 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.920863 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.922648 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.923362 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.923432 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.923482 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.923790 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.923849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260928 08:58:50.924093 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.919101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43643 (local address 127.0.0.1:51118)
0928 08:58:50.919282 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.919291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.919371 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.919546 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.919549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.919565 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:50.919635 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.919637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.920943 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.920944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.922499 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.922501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.923513 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.923513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.923577 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.923582 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.923630 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.923673 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":44,"thread_start_us":50,"threads_started":1}
I20260928 08:58:50.924458 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 5f556c2a40e14d6b9705b3127f68fe98 (DEFAULT_TABLE table=test-table [id=fa6002f911d04606a7d27803136d8bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:50.924719 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 5f556c2a40e14d6b9705b3127f68fe98 (DEFAULT_TABLE table=test-table [id=fa6002f911d04606a7d27803136d8bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:50.924861 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f556c2a40e14d6b9705b3127f68fe98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:50.925053 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f556c2a40e14d6b9705b3127f68fe98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:50.924976 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 5f556c2a40e14d6b9705b3127f68fe98 (DEFAULT_TABLE table=test-table [id=fa6002f911d04606a7d27803136d8bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:50.925269 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f556c2a40e14d6b9705b3127f68fe98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:50.927610 2621 tablet_bootstrap.cc:492] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Bootstrap starting.
I20260928 08:58:50.928365 2621 tablet_bootstrap.cc:654] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:50.928673 2623 tablet_bootstrap.cc:492] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Bootstrap starting.
I20260928 08:58:50.928891 2622 tablet_bootstrap.cc:492] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Bootstrap starting.
I20260928 08:58:50.928889 2621 log.cc:824] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:50.929484 2622 tablet_bootstrap.cc:654] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:50.929654 2621 tablet_bootstrap.cc:492] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: No bootstrap required, opened a new log
I20260928 08:58:50.929724 2621 ts_tablet_manager.cc:1397] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:50.929813 2622 log.cc:824] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:50.930577 2623 tablet_bootstrap.cc:654] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:50.930893 2623 log.cc:824] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:50.931440 2622 tablet_bootstrap.cc:492] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: No bootstrap required, opened a new log
I20260928 08:58:50.931507 2622 ts_tablet_manager.cc:1397] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:58:50.931538 2623 tablet_bootstrap.cc:492] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: No bootstrap required, opened a new log
I20260928 08:58:50.931608 2623 ts_tablet_manager.cc:1397] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:58:50.931908 2621 raft_consensus.cc:348] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.932077 2621 raft_consensus.cc:374] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:50.932114 2621 raft_consensus.cc:729] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084d19799cdf4f90b43f0553751bfcf1, State: Initialized, Role: FOLLOWER
I20260928 08:58:50.932231 2621 consensus_queue.cc:260] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.932488 2621 ts_tablet_manager.cc:1428] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:58:50.932512 2601 heartbeater.cc:503] Master 127.0.98.62:45427 was elected leader, sending a full tablet report...
I20260928 08:58:50.933386 2623 raft_consensus.cc:348] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.933555 2623 raft_consensus.cc:374] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:50.933599 2623 raft_consensus.cc:729] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f6550b2451441e9d70cbb6555babb7, State: Initialized, Role: FOLLOWER
I20260928 08:58:50.933710 2623 consensus_queue.cc:260] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.933645 2622 raft_consensus.cc:348] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.933799 2622 raft_consensus.cc:374] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:50.933874 2622 raft_consensus.cc:729] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Initialized, Role: FOLLOWER
I20260928 08:58:50.933981 2623 ts_tablet_manager.cc:1428] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:58:50.934002 2461 heartbeater.cc:503] Master 127.0.98.62:45427 was elected leader, sending a full tablet report...
I20260928 08:58:50.934015 2622 consensus_queue.cc:260] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.934264 2622 ts_tablet_manager.cc:1428] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:58:50.934370 2321 heartbeater.cc:503] Master 127.0.98.62:45427 was elected leader, sending a full tablet report...
I20260928 08:58:50.959369 2629 raft_consensus.cc:482] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:50.959473 2629 raft_consensus.cc:504] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.959736 2629 leader_election.cc:288] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549), 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643)
I20260928 08:58:50.962479 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.960037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41623 (local address 127.0.98.3:43549)
0928 08:58:50.960100 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.960102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.960134 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:58:50.960160 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.960161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.960173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.960223 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:50.960390 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.960915 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.961795 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.962138 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.962209 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.962219 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.962377 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.962408 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:58:50.962481 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.959968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43549 (local address 127.0.98.1:41623)
0928 08:58:50.960030 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.960043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.960098 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.960252 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.960253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.960255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:50.960316 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.960317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.960990 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.960991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.961709 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.961711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.962219 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.962219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.962292 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.962300 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.962369 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.962408 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260928 08:58:50.962750 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.960238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60265 (local address 127.0.98.2:43643)
0928 08:58:50.960278 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.960280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.960383 (+ 103us) server_negotiation.cc:407] Connection header received
0928 08:58:50.960467 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.960468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.960479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.960547 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:50.960713 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.961166 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.962044 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.962385 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.962466 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.962474 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:50.962641 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 08:58:50.962684 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:58:50.962781 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f556c2a40e14d6b9705b3127f68fe98" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1" is_pre_election: true
I20260928 08:58:50.962919 2555 raft_consensus.cc:2454] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 0.
I20260928 08:58:50.963039 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.960082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43643 (local address 127.0.98.1:60265)
0928 08:58:50.960322 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.960329 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.960390 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.960550 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.960551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.960552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:50.960623 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.960624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.961246 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.961247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.961957 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.961959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.962467 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.962468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.962547 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:50.962551 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:58:50.962882 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 08:58:50.962925 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
I20260928 08:58:50.963173 2208 leader_election.cc:302] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084d19799cdf4f90b43f0553751bfcf1, 6415149702a74917bbfc9eeb286128b3; no voters:
I20260928 08:58:50.963332 2629 raft_consensus.cc:2781] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:50.963397 2629 raft_consensus.cc:482] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:50.963428 2629 raft_consensus.cc:3037] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:50.963562 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f556c2a40e14d6b9705b3127f68fe98" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7" is_pre_election: true
I20260928 08:58:50.963725 2415 raft_consensus.cc:2454] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 0.
I20260928 08:58:50.964303 2629 raft_consensus.cc:504] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.964439 2629 leader_election.cc:288] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 election: Requested vote from peers 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549), 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643)
I20260928 08:58:50.964587 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f556c2a40e14d6b9705b3127f68fe98" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7"
I20260928 08:58:50.964576 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f556c2a40e14d6b9705b3127f68fe98" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1"
I20260928 08:58:50.964684 2415 raft_consensus.cc:3037] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:50.964690 2555 raft_consensus.cc:3037] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:50.965339 2415 raft_consensus.cc:2454] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 1.
I20260928 08:58:50.965483 2208 leader_election.cc:302] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6415149702a74917bbfc9eeb286128b3, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:50.965559 2629 raft_consensus.cc:2781] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:50.965574 2555 raft_consensus.cc:2454] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 1.
I20260928 08:58:50.965705 2629 raft_consensus.cc:686] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Running, Role: LEADER
I20260928 08:58:50.965791 2629 consensus_queue.cc:237] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } }
I20260928 08:58:50.966534 2117 catalog_manager.cc:5495] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6415149702a74917bbfc9eeb286128b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6415149702a74917bbfc9eeb286128b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } health_report { overall_health: UNKNOWN } } }
W20260928 08:58:50.987599 2462 tablet.cc:2369] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 08:58:50 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:51.003834 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.997321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.0.1:58816)
0928 08:58:50.997503 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:50.997518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:50.997603 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:50.997756 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:50.997757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:50.998068 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:50.998112 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.998114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.998743 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.998744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.999295 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:50.999297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.999650 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.999650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:50.999701 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.000182 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:51.000186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:51.001479 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:51.002797 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:51.002804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:51.002806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:51.002991 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:51.003187 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:51.003187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:51.003188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:51.003249 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:51.003473 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:51.003479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:51.003523 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.003598 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.003711 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260928 08:58:51.003836 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:50.997396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.0.98.62:45427)
0928 08:58:50.997478 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:50.997481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:50.997585 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:58:50.997682 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:50.997683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:50.997697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:50.997752 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:50.998195 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.998667 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.999377 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:50.999561 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:50.999651 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.001561 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:51.001563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:51.001563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:51.001570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:51.002709 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:51.003073 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:51.003074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:51.003074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:51.003128 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:51.003306 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:51.003307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:51.003308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:51.003412 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:51.003483 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.003631 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.003759 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":53}
I20260928 08:58:51.006112 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58816:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:58:51.006347 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:51.007709 2255 tablet_service.cc:1467] Processing CreateTablet for tablet fe00ec1dc11f491bb3f6f42aa6c719d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b781c5dcc05f44e1ba8746482a88b20f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:58:51.007745 2395 tablet_service.cc:1467] Processing CreateTablet for tablet fe00ec1dc11f491bb3f6f42aa6c719d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b781c5dcc05f44e1ba8746482a88b20f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:58:51.007772 2535 tablet_service.cc:1467] Processing CreateTablet for tablet fe00ec1dc11f491bb3f6f42aa6c719d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b781c5dcc05f44e1ba8746482a88b20f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:58:51.007890 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe00ec1dc11f491bb3f6f42aa6c719d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:51.007890 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe00ec1dc11f491bb3f6f42aa6c719d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:51.007908 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe00ec1dc11f491bb3f6f42aa6c719d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:51.009210 2623 tablet_bootstrap.cc:492] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: Bootstrap starting.
I20260928 08:58:51.009500 2621 tablet_bootstrap.cc:492] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: Bootstrap starting.
I20260928 08:58:51.009648 2622 tablet_bootstrap.cc:492] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: Bootstrap starting.
I20260928 08:58:51.010053 2621 tablet_bootstrap.cc:654] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:51.010053 2623 tablet_bootstrap.cc:654] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:51.010076 2622 tablet_bootstrap.cc:654] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:51.010690 2622 tablet_bootstrap.cc:492] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: No bootstrap required, opened a new log
I20260928 08:58:51.010732 2622 ts_tablet_manager.cc:1397] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:51.010885 2622 raft_consensus.cc:348] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.010962 2622 raft_consensus.cc:374] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:51.010972 2623 tablet_bootstrap.cc:492] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: No bootstrap required, opened a new log
I20260928 08:58:51.010987 2622 raft_consensus.cc:729] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Initialized, Role: FOLLOWER
I20260928 08:58:51.011026 2623 ts_tablet_manager.cc:1397] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:51.011034 2622 consensus_queue.cc:260] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.011118 2622 ts_tablet_manager.cc:1428] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:51.011173 2623 raft_consensus.cc:348] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.011245 2623 raft_consensus.cc:374] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:51.011270 2623 raft_consensus.cc:729] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f6550b2451441e9d70cbb6555babb7, State: Initialized, Role: FOLLOWER
I20260928 08:58:51.011256 2629 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.011314 2623 consensus_queue.cc:260] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.011344 2629 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:51.011404 2623 ts_tablet_manager.cc:1428] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:51.011546 2628 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.011618 2628 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:51.011931 2621 tablet_bootstrap.cc:492] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: No bootstrap required, opened a new log
I20260928 08:58:51.011979 2621 ts_tablet_manager.cc:1397] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:51.012132 2621 raft_consensus.cc:348] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.012200 2621 raft_consensus.cc:374] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:51.012223 2621 raft_consensus.cc:729] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084d19799cdf4f90b43f0553751bfcf1, State: Initialized, Role: FOLLOWER
I20260928 08:58:51.012277 2621 consensus_queue.cc:260] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.012368 2621 ts_tablet_manager.cc:1428] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:51.012459 2627 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.012567 2627 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 08:58:51.083015 2322 tablet.cc:2369] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:58:51.131889 2602 tablet.cc:2369] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:58:51.348610 2628 raft_consensus.cc:482] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:51.348749 2628 raft_consensus.cc:504] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.349115 2628 leader_election.cc:288] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429), 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:51.351998 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.349440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53875 (local address 127.0.98.3:43549)
0928 08:58:51.349508 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.349510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.349522 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:58:51.349582 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.349583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.349593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.349643 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:51.349804 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.350310 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.351300 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.351651 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.351704 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.351712 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.351886 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.351926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:58:51.352025 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.349341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44877 (local address 127.0.98.1:41429)
0928 08:58:51.349391 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.349394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.349518 (+ 124us) server_negotiation.cc:407] Connection header received
0928 08:58:51.349601 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.349602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.349618 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.349689 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:51.349856 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.350336 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.351330 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.351676 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.351761 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.351771 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.351922 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.351961 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:58:51.352033 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.349302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43549 (local address 127.0.98.2:53875)
0928 08:58:51.349449 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.349460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.349519 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.349665 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.349666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.349668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:51.349730 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.349732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.350384 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.350385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.351206 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.351208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.351729 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.351731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.351808 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.351812 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.351863 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.351918 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260928 08:58:51.352043 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.349282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41429 (local address 127.0.98.2:44877)
0928 08:58:51.349425 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.349442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.349529 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.349700 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.349701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.349703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:51.349784 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.349786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.350430 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.350431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.351204 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.351206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.351776 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.351777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.351831 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.351835 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.351906 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.351943 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":40,"thread_start_us":91,"threads_started":1}
I20260928 08:58:51.352346 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe00ec1dc11f491bb3f6f42aa6c719d2" candidate_uuid: "90f6550b2451441e9d70cbb6555babb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6415149702a74917bbfc9eeb286128b3" is_pre_election: true
I20260928 08:58:51.352481 2275 raft_consensus.cc:2454] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f6550b2451441e9d70cbb6555babb7 in term 0.
I20260928 08:58:51.352517 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe00ec1dc11f491bb3f6f42aa6c719d2" candidate_uuid: "90f6550b2451441e9d70cbb6555babb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1" is_pre_election: true
I20260928 08:58:51.352610 2555 raft_consensus.cc:2454] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f6550b2451441e9d70cbb6555babb7 in term 0.
I20260928 08:58:51.352699 2347 leader_election.cc:302] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6415149702a74917bbfc9eeb286128b3, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:51.352834 2628 raft_consensus.cc:2781] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:51.352876 2628 raft_consensus.cc:482] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:51.352905 2628 raft_consensus.cc:3037] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:51.353572 2628 raft_consensus.cc:504] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.353715 2628 leader_election.cc:288] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [CANDIDATE]: Term 1 election: Requested vote from peers 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429), 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:51.353904 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe00ec1dc11f491bb3f6f42aa6c719d2" candidate_uuid: "90f6550b2451441e9d70cbb6555babb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6415149702a74917bbfc9eeb286128b3"
I20260928 08:58:51.353945 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe00ec1dc11f491bb3f6f42aa6c719d2" candidate_uuid: "90f6550b2451441e9d70cbb6555babb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1"
I20260928 08:58:51.353988 2275 raft_consensus.cc:3037] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:51.353996 2555 raft_consensus.cc:3037] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:51.354614 2555 raft_consensus.cc:2454] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f6550b2451441e9d70cbb6555babb7 in term 1.
I20260928 08:58:51.354621 2275 raft_consensus.cc:2454] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f6550b2451441e9d70cbb6555babb7 in term 1.
I20260928 08:58:51.354773 2348 leader_election.cc:302] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084d19799cdf4f90b43f0553751bfcf1, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:51.354864 2628 raft_consensus.cc:2781] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:51.354985 2628 raft_consensus.cc:686] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [term 1 LEADER]: Becoming Leader. State: Replica: 90f6550b2451441e9d70cbb6555babb7, State: Running, Role: LEADER
I20260928 08:58:51.355079 2628 consensus_queue.cc:237] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.355468 2644 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: TxnStatusTablet state changed. Reason: New leader 90f6550b2451441e9d70cbb6555babb7. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.355554 2644 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:58:51.355825 2646 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:58:51.355942 2646 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:58:51.355894 2118 catalog_manager.cc:5495] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } health_report { overall_health: UNKNOWN } } }
I20260928 08:58:51.365571 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.362973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.0.98.2:43643)
0928 08:58:51.363039 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.363042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.363073 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:58:51.363181 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.363181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.363198 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.363255 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.363373 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.363847 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.364743 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.364994 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.365077 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.365260 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.365361 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.365447 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.365475 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.365497 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:58:51.365590 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.362923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43643 (local address 127.0.0.1:51128)
0928 08:58:51.363015 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.363038 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.363100 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.363265 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.363266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.363267 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:51.363315 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.363317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.363937 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.363938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.364677 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.364679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.365078 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.365079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.365143 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.365162 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.365424 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.365432 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.365473 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.365519 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:58:51.367189 2555 raft_consensus.cc:1261] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Refusing update from remote peer 90f6550b2451441e9d70cbb6555babb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:51.367234 2275 raft_consensus.cc:1261] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Refusing update from remote peer 90f6550b2451441e9d70cbb6555babb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:51.367486 2644 consensus_queue.cc:1035] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:51.367594 2628 consensus_queue.cc:1035] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:51.368582 2629 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: TxnStatusTablet state changed. Reason: New leader 90f6550b2451441e9d70cbb6555babb7. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.368695 2629 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:51.368686 2627 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: TxnStatusTablet state changed. Reason: New leader 90f6550b2451441e9d70cbb6555babb7. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.368785 2627 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:51.369549 2628 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.369640 2628 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:58:51.369649 2627 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.369724 2627 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 084d19799cdf4f90b43f0553751bfcf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:51.370375 2649 mvcc.cc:205] Tried to move back new op lower bound from 7334239974877622272 to 7334239974830960640. Current Snapshot: MvccSnapshot[applied={T|T < 7334239974877622272 or (T in {7334239974877622272})}]
I20260928 08:58:51.373045 2651 mvcc.cc:205] Tried to move back new op lower bound from 7334239974877622272 to 7334239974830960640. Current Snapshot: MvccSnapshot[applied={T|T < 7334239974877622272}]
I20260928 08:58:51.373996 2628 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.374084 2628 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 90f6550b2451441e9d70cbb6555babb7: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:58:51.374312 2650 mvcc.cc:205] Tried to move back new op lower bound from 7334239974877622272 to 7334239974830960640. Current Snapshot: MvccSnapshot[applied={T|T < 7334239974877622272}]
I20260928 08:58:51.374408 2629 tablet_replica.cc:439] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90f6550b2451441e9d70cbb6555babb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } } }
I20260928 08:58:51.374482 2629 tablet_replica.cc:442] T fe00ec1dc11f491bb3f6f42aa6c719d2 P 6415149702a74917bbfc9eeb286128b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:58:51.374557 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.371483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.98.1:41429)
0928 08:58:51.371525 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.371527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.371536 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:51.371604 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.371604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.371622 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.371684 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.371818 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.372374 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.373480 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.373701 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.373776 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.373958 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.374078 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.374160 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.374414 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.374438 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:58:51.377522 2629 consensus_queue.cc:1035] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:51.379660 2629 consensus_queue.cc:1035] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:51.380098 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.376967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45859 (local address 127.0.98.2:43643)
0928 08:58:51.377026 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.377028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.377266 (+ 238us) server_negotiation.cc:407] Connection header received
0928 08:58:51.377330 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.377331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.377348 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.377401 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.377537 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.378119 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.379233 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.379504 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.379545 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.379811 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.379846 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.379875 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.380004 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.380026 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 08:58:51.380568 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.376896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43643 (local address 127.0.98.1:45859)
0928 08:58:51.377201 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.377214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.377278 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.377404 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.377405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.377407 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:51.377470 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.377472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.378192 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.378194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.379132 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.379134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.379605 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.379614 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.379683 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.379698 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.379921 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.379922 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.380360 (+ 438us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.380477 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":97,"threads_started":1}
I20260928 08:58:51.390082 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.387110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41429 (local address 127.0.98.2:58111)
0928 08:58:51.387363 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.387374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.387439 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.387573 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.387574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.387576 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:51.387629 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.387631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.388491 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.388492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.389293 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.389295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.389732 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.389733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.389776 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.389788 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.389941 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.389942 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.389988 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.390016 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":73,"threads_started":1}
I20260928 08:58:51.390081 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.387169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58111 (local address 127.0.98.1:41429)
0928 08:58:51.387211 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.387213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.387424 (+ 211us) server_negotiation.cc:407] Connection header received
0928 08:58:51.387504 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.387505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.387517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.387571 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.387688 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.388417 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.389370 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.389663 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.389734 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.389854 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.389886 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.389942 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.389995 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.390012 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:58:51.398993 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:51.399096 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:51.399793 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Sending DeleteTablet for 3 replicas of tablet 5f556c2a40e14d6b9705b3127f68fe98
I20260928 08:58:51.400054 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f556c2a40e14d6b9705b3127f68fe98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57536
I20260928 08:58:51.400128 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f556c2a40e14d6b9705b3127f68fe98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51118
I20260928 08:58:51.400449 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f556c2a40e14d6b9705b3127f68fe98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260928 08:58:51.400686 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:51.400815 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:51.401858 2255 tablet_service.cc:1467] Processing CreateTablet for tablet afdc7439372a42c7b4bf28c46968c6c6 (DEFAULT_TABLE table=test-table [id=e1b864ae410b41e38020bdf84ce0c154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:51.401901 2535 tablet_service.cc:1467] Processing CreateTablet for tablet afdc7439372a42c7b4bf28c46968c6c6 (DEFAULT_TABLE table=test-table [id=e1b864ae410b41e38020bdf84ce0c154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:51.401988 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc7439372a42c7b4bf28c46968c6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:51.401981 2395 tablet_service.cc:1467] Processing CreateTablet for tablet afdc7439372a42c7b4bf28c46968c6c6 (DEFAULT_TABLE table=test-table [id=e1b864ae410b41e38020bdf84ce0c154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:51.402019 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc7439372a42c7b4bf28c46968c6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:51.402070 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc7439372a42c7b4bf28c46968c6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:51.402974 2622 tablet_bootstrap.cc:492] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Bootstrap starting.
I20260928 08:58:51.403313 2623 tablet_bootstrap.cc:492] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Bootstrap starting.
I20260928 08:58:51.403313 2621 tablet_bootstrap.cc:492] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Bootstrap starting.
I20260928 08:58:51.403371 2622 tablet_bootstrap.cc:654] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:51.403689 2621 tablet_bootstrap.cc:654] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:51.403689 2623 tablet_bootstrap.cc:654] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:51.403944 2622 tablet_bootstrap.cc:492] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: No bootstrap required, opened a new log
I20260928 08:58:51.403995 2622 ts_tablet_manager.cc:1397] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:51.404120 2622 raft_consensus.cc:348] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.404182 2622 raft_consensus.cc:374] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:51.404202 2622 raft_consensus.cc:729] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Initialized, Role: FOLLOWER
I20260928 08:58:51.404244 2622 consensus_queue.cc:260] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.404338 2622 ts_tablet_manager.cc:1428] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:51.405018 2623 tablet_bootstrap.cc:492] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: No bootstrap required, opened a new log
I20260928 08:58:51.405072 2623 ts_tablet_manager.cc:1397] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:51.405200 2621 tablet_bootstrap.cc:492] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: No bootstrap required, opened a new log
I20260928 08:58:51.405237 2621 ts_tablet_manager.cc:1397] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:58:51.405205 2623 raft_consensus.cc:348] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.405267 2623 raft_consensus.cc:374] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:51.405292 2623 raft_consensus.cc:729] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f6550b2451441e9d70cbb6555babb7, State: Initialized, Role: FOLLOWER
I20260928 08:58:51.405337 2623 consensus_queue.cc:260] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.405396 2621 raft_consensus.cc:348] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.405452 2621 raft_consensus.cc:374] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:51.405468 2621 raft_consensus.cc:729] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084d19799cdf4f90b43f0553751bfcf1, State: Initialized, Role: FOLLOWER
I20260928 08:58:51.405504 2621 consensus_queue.cc:260] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.405594 2621 ts_tablet_manager.cc:1428] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:51.405890 2623 ts_tablet_manager.cc:1428] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:51.406301 2672 tablet_replica.cc:330] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: stopping tablet replica
I20260928 08:58:51.406428 2672 raft_consensus.cc:2229] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:58:51.406548 2672 raft_consensus.cc:2258] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:51.406975 2672 ts_tablet_manager.cc:1905] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:51.408593 2670 tablet_replica.cc:330] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: stopping tablet replica
I20260928 08:58:51.408588 2672 ts_tablet_manager.cc:1918] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:51.408676 2672 log.cc:1196] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/5f556c2a40e14d6b9705b3127f68fe98
I20260928 08:58:51.408711 2670 raft_consensus.cc:2229] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:51.408775 2670 raft_consensus.cc:2258] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:51.408936 2672 ts_tablet_manager.cc:1939] T 5f556c2a40e14d6b9705b3127f68fe98 P 6415149702a74917bbfc9eeb286128b3: Deleting consensus metadata
I20260928 08:58:51.409351 2103 catalog_manager.cc:4841] TS 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429): tablet 5f556c2a40e14d6b9705b3127f68fe98 (table test-table [id=fa6002f911d04606a7d27803136d8bb4]) successfully deleted
I20260928 08:58:51.412787 2671 tablet_replica.cc:330] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: stopping tablet replica
I20260928 08:58:51.412952 2671 raft_consensus.cc:2229] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:51.413064 2671 raft_consensus.cc:2258] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:51.413156 2670 ts_tablet_manager.cc:1905] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:51.413579 2671 ts_tablet_manager.cc:1905] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:51.414907 2670 ts_tablet_manager.cc:1918] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:51.415020 2670 log.cc:1196] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/5f556c2a40e14d6b9705b3127f68fe98
I20260928 08:58:51.415282 2670 ts_tablet_manager.cc:1939] T 5f556c2a40e14d6b9705b3127f68fe98 P 084d19799cdf4f90b43f0553751bfcf1: Deleting consensus metadata
I20260928 08:58:51.415447 2671 ts_tablet_manager.cc:1918] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:51.415546 2671 log.cc:1196] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/5f556c2a40e14d6b9705b3127f68fe98
I20260928 08:58:51.415697 2104 catalog_manager.cc:4841] TS 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549): tablet 5f556c2a40e14d6b9705b3127f68fe98 (table test-table [id=fa6002f911d04606a7d27803136d8bb4]) successfully deleted
I20260928 08:58:51.415908 2671 ts_tablet_manager.cc:1939] T 5f556c2a40e14d6b9705b3127f68fe98 P 90f6550b2451441e9d70cbb6555babb7: Deleting consensus metadata
I20260928 08:58:51.416309 2104 catalog_manager.cc:4841] TS 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643): tablet 5f556c2a40e14d6b9705b3127f68fe98 (table test-table [id=fa6002f911d04606a7d27803136d8bb4]) successfully deleted
I20260928 08:58:51.427928 2627 raft_consensus.cc:482] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:51.428025 2627 raft_consensus.cc:504] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.428346 2627 leader_election.cc:288] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429), 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643)
I20260928 08:58:51.432333 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.428712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37225 (local address 127.0.98.2:43643)
0928 08:58:51.428753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.428755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.428878 (+ 123us) server_negotiation.cc:407] Connection header received
0928 08:58:51.428974 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.428975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.428983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.429011 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:51.429222 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.429949 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.431406 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.431960 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.432007 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.432017 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.432229 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.432263 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:58:51.432612 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.428686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43219 (local address 127.0.98.1:41429)
0928 08:58:51.428728 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.428730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.428859 (+ 129us) server_negotiation.cc:407] Connection header received
0928 08:58:51.428959 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.428960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.428982 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.429011 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:58:51.429218 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.429957 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.431698 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.432257 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.432301 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.432309 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.432513 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.432546 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:58:51.432955 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.428530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43643 (local address 127.0.98.3:37225)
0928 08:58:51.428814 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.428824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.428904 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.429071 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.429071 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.429073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:51.429158 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.429159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.430052 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.430053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.431291 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.431294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.432074 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.432075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.432155 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.432158 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.432665 (+ 507us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.432831 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":63,"thread_start_us":46,"threads_started":1}
I20260928 08:58:51.432974 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.428502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41429 (local address 127.0.98.3:43219)
0928 08:58:51.428797 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.428807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.428887 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.429073 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.429073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.429075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:58:51.429149 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.429151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.430078 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.430078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.431596 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.431599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.432366 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.432367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.432432 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.432435 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.432747 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.432854 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":102,"thread_start_us":52,"threads_started":1}
I20260928 08:58:51.433317 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc7439372a42c7b4bf28c46968c6c6" candidate_uuid: "084d19799cdf4f90b43f0553751bfcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7" is_pre_election: true
I20260928 08:58:51.433337 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc7439372a42c7b4bf28c46968c6c6" candidate_uuid: "084d19799cdf4f90b43f0553751bfcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6415149702a74917bbfc9eeb286128b3" is_pre_election: true
I20260928 08:58:51.433408 2415 raft_consensus.cc:2454] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 084d19799cdf4f90b43f0553751bfcf1 in term 0.
I20260928 08:58:51.433414 2275 raft_consensus.cc:2454] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 084d19799cdf4f90b43f0553751bfcf1 in term 0.
I20260928 08:58:51.433565 2487 leader_election.cc:302] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084d19799cdf4f90b43f0553751bfcf1, 6415149702a74917bbfc9eeb286128b3; no voters:
I20260928 08:58:51.433673 2627 raft_consensus.cc:2781] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:51.433701 2627 raft_consensus.cc:482] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:51.433722 2627 raft_consensus.cc:3037] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:51.434542 2627 raft_consensus.cc:504] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.434672 2627 leader_election.cc:288] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [CANDIDATE]: Term 1 election: Requested vote from peers 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429), 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643)
I20260928 08:58:51.434827 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc7439372a42c7b4bf28c46968c6c6" candidate_uuid: "084d19799cdf4f90b43f0553751bfcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7"
I20260928 08:58:51.434839 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc7439372a42c7b4bf28c46968c6c6" candidate_uuid: "084d19799cdf4f90b43f0553751bfcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6415149702a74917bbfc9eeb286128b3"
I20260928 08:58:51.434892 2415 raft_consensus.cc:3037] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:51.434898 2275 raft_consensus.cc:3037] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:51.435585 2415 raft_consensus.cc:2454] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 084d19799cdf4f90b43f0553751bfcf1 in term 1.
I20260928 08:58:51.435648 2275 raft_consensus.cc:2454] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 084d19799cdf4f90b43f0553751bfcf1 in term 1.
I20260928 08:58:51.435775 2488 leader_election.cc:302] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084d19799cdf4f90b43f0553751bfcf1, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:51.435907 2627 raft_consensus.cc:2781] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:51.436028 2627 raft_consensus.cc:686] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 LEADER]: Becoming Leader. State: Replica: 084d19799cdf4f90b43f0553751bfcf1, State: Running, Role: LEADER
I20260928 08:58:51.436095 2627 consensus_queue.cc:237] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:51.436774 2117 catalog_manager.cc:5495] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "084d19799cdf4f90b43f0553751bfcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } health_report { overall_health: HEALTHY } } }
I20260928 08:58:51.445096 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.441950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.0.98.3:43549)
0928 08:58:51.441995 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.441997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.442030 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:58:51.442126 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.442127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.442152 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.442234 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.442372 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.443109 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.444176 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.444480 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.444552 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.444727 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.444844 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.444916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.444974 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.444994 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:58:51.450832 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.447893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56745 (local address 127.0.98.2:43643)
0928 08:58:51.447929 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.447931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.447954 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:58:51.448017 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.448018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.448027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.448078 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.448233 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.448974 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.450029 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.450326 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.450397 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.450588 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.450621 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.450669 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.450738 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.450756 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:58:51.451071 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.447726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43643 (local address 127.0.98.3:56745)
0928 08:58:51.447892 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.447904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.447947 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.448090 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.448091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.448093 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:51.448166 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.448168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.449056 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.449057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.449940 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.449943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.450409 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.450411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.450478 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.450492 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.450684 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.450686 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.450738 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.450781 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 08:58:51.453585 2415 raft_consensus.cc:1261] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Refusing update from remote peer 084d19799cdf4f90b43f0553751bfcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:51.453586 2275 raft_consensus.cc:1261] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Refusing update from remote peer 084d19799cdf4f90b43f0553751bfcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:51.453796 2627 consensus_queue.cc:1035] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:51.453902 2678 consensus_queue.cc:1035] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:51.462548 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.459650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33079 (local address 127.0.98.3:43549)
0928 08:58:51.459686 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:51.459687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:51.459696 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:58:51.459752 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:51.459752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:51.459762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:51.459815 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:51.460021 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.460740 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.461790 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.462063 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.462128 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.462300 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.462333 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.462386 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:51.462457 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 08:58:51.462474 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:58:51.462574 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:51.459575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43549 (local address 127.0.98.2:33079)
0928 08:58:51.459647 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:51.459659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:51.459698 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:51.459864 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:51.459865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:51.459866 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:58:51.459906 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.459907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.460832 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.460833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.461692 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:51.461695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:51.462162 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:51.462163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:51.462216 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:51.462221 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:58:51.462405 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:58:51.462407 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:58:51.462456 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:58:51.462500 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:58:51.463059 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:58:52.986248 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:52.986364 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:52.987118 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Sending DeleteTablet for 3 replicas of tablet afdc7439372a42c7b4bf28c46968c6c6
I20260928 08:58:52.987392 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet afdc7439372a42c7b4bf28c46968c6c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51118
I20260928 08:58:52.987406 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet afdc7439372a42c7b4bf28c46968c6c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57536
I20260928 08:58:52.987582 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet afdc7439372a42c7b4bf28c46968c6c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260928 08:58:52.987763 2709 tablet_replica.cc:330] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: stopping tablet replica
I20260928 08:58:52.987814 2709 raft_consensus.cc:2229] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:52.987847 2709 raft_consensus.cc:2258] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:52.988096 2709 ts_tablet_manager.cc:1905] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:52.989514 2707 tablet_replica.cc:330] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: stopping tablet replica
I20260928 08:58:52.989615 2707 raft_consensus.cc:2229] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:52.989696 2707 raft_consensus.cc:2258] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:52.989908 2709 ts_tablet_manager.cc:1918] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:58:52.989966 2707 ts_tablet_manager.cc:1905] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:52.990015 2709 log.cc:1196] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/afdc7439372a42c7b4bf28c46968c6c6
I20260928 08:58:52.990244 2709 ts_tablet_manager.cc:1939] T afdc7439372a42c7b4bf28c46968c6c6 P 6415149702a74917bbfc9eeb286128b3: Deleting consensus metadata
I20260928 08:58:52.990645 2103 catalog_manager.cc:4841] TS 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429): tablet afdc7439372a42c7b4bf28c46968c6c6 (table test-table [id=e1b864ae410b41e38020bdf84ce0c154]) successfully deleted
I20260928 08:58:52.993549 2707 ts_tablet_manager.cc:1918] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:58:52.993623 2707 log.cc:1196] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/afdc7439372a42c7b4bf28c46968c6c6
I20260928 08:58:52.993785 2708 tablet_replica.cc:330] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: stopping tablet replica
I20260928 08:58:52.993841 2707 ts_tablet_manager.cc:1939] T afdc7439372a42c7b4bf28c46968c6c6 P 90f6550b2451441e9d70cbb6555babb7: Deleting consensus metadata
I20260928 08:58:52.993862 2708 raft_consensus.cc:2229] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:58:52.993957 2708 raft_consensus.cc:2258] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:52.994222 2708 ts_tablet_manager.cc:1905] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:52.994231 2104 catalog_manager.cc:4841] TS 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643): tablet afdc7439372a42c7b4bf28c46968c6c6 (table test-table [id=e1b864ae410b41e38020bdf84ce0c154]) successfully deleted
I20260928 08:58:52.995529 2708 ts_tablet_manager.cc:1918] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:58:52.995599 2708 log.cc:1196] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/afdc7439372a42c7b4bf28c46968c6c6
I20260928 08:58:52.995816 2708 ts_tablet_manager.cc:1939] T afdc7439372a42c7b4bf28c46968c6c6 P 084d19799cdf4f90b43f0553751bfcf1: Deleting consensus metadata
I20260928 08:58:52.996120 2104 catalog_manager.cc:4841] TS 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549): tablet afdc7439372a42c7b4bf28c46968c6c6 (table test-table [id=e1b864ae410b41e38020bdf84ce0c154]) successfully deleted
Sep 28 08:58:52 dist-test-slave-286n krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:58:53 dist-test-slave-286n krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585932, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:53.010596 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:53.003559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.0.98.62:45427)
0928 08:58:53.003721 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:53.003723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:53.003748 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:58:53.003840 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:53.003841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:53.003852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:53.003912 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:53.004426 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:53.004984 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:53.005670 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:53.005859 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:53.005919 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:53.008130 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:53.008132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:53.008132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:53.008139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:53.009519 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:53.009855 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:53.009857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:53.009857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:53.009913 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:53.010093 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:53.010094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:53.010094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:53.010205 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:53.010306 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:53.010401 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 08:58:53.010481 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260928 08:58:53.011974 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:53.012140 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:53.013460 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 2731e419b0e44696b2745e75225b54e1 (DEFAULT_TABLE table=test-table [id=bbc19bb48b6646139888806a26fc111d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:53.013460 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 2731e419b0e44696b2745e75225b54e1 (DEFAULT_TABLE table=test-table [id=bbc19bb48b6646139888806a26fc111d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:53.013584 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2731e419b0e44696b2745e75225b54e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:53.013584 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2731e419b0e44696b2745e75225b54e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:53.014942 2719 tablet_bootstrap.cc:492] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Bootstrap starting.
I20260928 08:58:53.014968 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 2731e419b0e44696b2745e75225b54e1 (DEFAULT_TABLE table=test-table [id=bbc19bb48b6646139888806a26fc111d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:53.015110 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2731e419b0e44696b2745e75225b54e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:53.015277 2719 tablet_bootstrap.cc:654] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:53.015523 2718 tablet_bootstrap.cc:492] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Bootstrap starting.
I20260928 08:58:53.015828 2718 tablet_bootstrap.cc:654] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:53.015863 2719 tablet_bootstrap.cc:492] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: No bootstrap required, opened a new log
I20260928 08:58:53.015906 2719 ts_tablet_manager.cc:1397] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:53.016021 2719 raft_consensus.cc:348] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.016137 2719 raft_consensus.cc:374] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:53.016155 2719 raft_consensus.cc:729] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f6550b2451441e9d70cbb6555babb7, State: Initialized, Role: FOLLOWER
I20260928 08:58:53.016212 2719 consensus_queue.cc:260] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.016321 2719 ts_tablet_manager.cc:1428] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:53.016378 2718 tablet_bootstrap.cc:492] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: No bootstrap required, opened a new log
I20260928 08:58:53.016436 2718 ts_tablet_manager.cc:1397] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:53.016552 2718 raft_consensus.cc:348] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.016654 2718 raft_consensus.cc:374] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:53.016682 2718 raft_consensus.cc:729] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084d19799cdf4f90b43f0553751bfcf1, State: Initialized, Role: FOLLOWER
I20260928 08:58:53.016736 2718 consensus_queue.cc:260] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.016870 2718 ts_tablet_manager.cc:1428] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:53.017472 2722 tablet_bootstrap.cc:492] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Bootstrap starting.
I20260928 08:58:53.017796 2722 tablet_bootstrap.cc:654] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:53.018363 2722 tablet_bootstrap.cc:492] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: No bootstrap required, opened a new log
I20260928 08:58:53.018417 2722 ts_tablet_manager.cc:1397] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:53.018543 2722 raft_consensus.cc:348] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.018608 2722 raft_consensus.cc:374] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:53.018631 2722 raft_consensus.cc:729] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Initialized, Role: FOLLOWER
I20260928 08:58:53.018683 2722 consensus_queue.cc:260] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.018920 2722 ts_tablet_manager.cc:1428] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:53.166388 2724 raft_consensus.cc:482] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:53.166510 2724 raft_consensus.cc:504] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.166716 2724 leader_election.cc:288] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643), 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:53.166936 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2731e419b0e44696b2745e75225b54e1" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7" is_pre_election: true
I20260928 08:58:53.166952 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2731e419b0e44696b2745e75225b54e1" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1" is_pre_election: true
I20260928 08:58:53.167033 2415 raft_consensus.cc:2454] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 0.
I20260928 08:58:53.167068 2555 raft_consensus.cc:2454] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 0.
I20260928 08:58:53.167220 2208 leader_election.cc:302] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6415149702a74917bbfc9eeb286128b3, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:53.167358 2724 raft_consensus.cc:2781] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:53.167403 2724 raft_consensus.cc:482] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:53.167419 2724 raft_consensus.cc:3037] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:53.168161 2724 raft_consensus.cc:504] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.168273 2724 leader_election.cc:288] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 election: Requested vote from peers 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643), 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:53.168512 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2731e419b0e44696b2745e75225b54e1" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1"
I20260928 08:58:53.168489 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2731e419b0e44696b2745e75225b54e1" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7"
I20260928 08:58:53.168574 2555 raft_consensus.cc:3037] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:53.168593 2415 raft_consensus.cc:3037] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:53.169222 2555 raft_consensus.cc:2454] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 1.
I20260928 08:58:53.169235 2415 raft_consensus.cc:2454] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 1.
I20260928 08:58:53.169415 2208 leader_election.cc:302] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084d19799cdf4f90b43f0553751bfcf1, 6415149702a74917bbfc9eeb286128b3; no voters:
I20260928 08:58:53.169531 2724 raft_consensus.cc:2781] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:53.169692 2724 raft_consensus.cc:686] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Running, Role: LEADER
I20260928 08:58:53.169739 2724 consensus_queue.cc:237] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.170454 2118 catalog_manager.cc:5495] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6415149702a74917bbfc9eeb286128b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6415149702a74917bbfc9eeb286128b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } health_report { overall_health: UNKNOWN } } }
I20260928 08:58:53.199107 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:53.196804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.98.1:41429)
0928 08:58:53.196958 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:53.196960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:53.196966 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:58:53.196987 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:53.196987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:53.196998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:53.197079 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:58:53.197190 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:53.197674 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:53.198439 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:53.198649 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:53.198718 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:53.198838 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:58:53.198871 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:58:53.198934 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:53.198988 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 08:58:53.199009 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 08:58:53.201380 2555 raft_consensus.cc:1261] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Refusing update from remote peer 6415149702a74917bbfc9eeb286128b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:53.201414 2415 raft_consensus.cc:1261] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Refusing update from remote peer 6415149702a74917bbfc9eeb286128b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:53.201649 2724 consensus_queue.cc:1035] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:53.201746 2725 consensus_queue.cc:1035] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:53.211485 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:53.211593 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:53.212224 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Sending DeleteTablet for 3 replicas of tablet 2731e419b0e44696b2745e75225b54e1
I20260928 08:58:53.212489 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 2731e419b0e44696b2745e75225b54e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57536
I20260928 08:58:53.212558 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 2731e419b0e44696b2745e75225b54e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260928 08:58:53.212546 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 2731e419b0e44696b2745e75225b54e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51118
I20260928 08:58:53.212598 2708 tablet_replica.cc:330] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: stopping tablet replica
I20260928 08:58:53.212664 2709 tablet_replica.cc:330] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: stopping tablet replica
I20260928 08:58:53.212666 2708 raft_consensus.cc:2229] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:53.212703 2708 raft_consensus.cc:2258] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:53.212714 2709 raft_consensus.cc:2229] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:58:53.212769 2709 raft_consensus.cc:2258] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:53.212779 2707 tablet_replica.cc:330] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: stopping tablet replica
I20260928 08:58:53.212843 2707 raft_consensus.cc:2229] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:53.212885 2707 raft_consensus.cc:2258] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:53.212831 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:58:53.212980 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:58:53.213201 2709 ts_tablet_manager.cc:1905] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:53.213488 2708 ts_tablet_manager.cc:1905] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:53.214794 2707 ts_tablet_manager.cc:1905] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:53.214910 2709 ts_tablet_manager.cc:1918] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:53.214968 2709 log.cc:1196] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/2731e419b0e44696b2745e75225b54e1
I20260928 08:58:53.215167 2709 ts_tablet_manager.cc:1939] T 2731e419b0e44696b2745e75225b54e1 P 6415149702a74917bbfc9eeb286128b3: Deleting consensus metadata
I20260928 08:58:53.215503 2708 ts_tablet_manager.cc:1918] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:53.215557 2708 log.cc:1196] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/2731e419b0e44696b2745e75225b54e1
I20260928 08:58:53.215621 2103 catalog_manager.cc:4841] TS 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429): tablet 2731e419b0e44696b2745e75225b54e1 (table test-table [id=bbc19bb48b6646139888806a26fc111d]) successfully deleted
I20260928 08:58:53.215791 2708 ts_tablet_manager.cc:1939] T 2731e419b0e44696b2745e75225b54e1 P 084d19799cdf4f90b43f0553751bfcf1: Deleting consensus metadata
I20260928 08:58:53.216125 2104 catalog_manager.cc:4841] TS 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549): tablet 2731e419b0e44696b2745e75225b54e1 (table test-table [id=bbc19bb48b6646139888806a26fc111d]) successfully deleted
I20260928 08:58:53.216223 2707 ts_tablet_manager.cc:1918] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:58:53.216264 2707 log.cc:1196] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/2731e419b0e44696b2745e75225b54e1
I20260928 08:58:53.216506 2707 ts_tablet_manager.cc:1939] T 2731e419b0e44696b2745e75225b54e1 P 90f6550b2451441e9d70cbb6555babb7: Deleting consensus metadata
I20260928 08:58:53.217142 2104 catalog_manager.cc:4841] TS 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643): tablet 2731e419b0e44696b2745e75225b54e1 (table test-table [id=bbc19bb48b6646139888806a26fc111d]) successfully deleted
I20260928 08:58:53.218046 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda (DEFAULT_TABLE table=test-table [id=521eb1cd39b94eaaaf4bd40332571ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:53.218077 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda (DEFAULT_TABLE table=test-table [id=521eb1cd39b94eaaaf4bd40332571ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:53.218164 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:53.218199 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:53.219455 2722 tablet_bootstrap.cc:492] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3: Bootstrap starting.
I20260928 08:58:53.219504 2719 tablet_bootstrap.cc:492] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Bootstrap starting.
I20260928 08:58:53.219535 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda (DEFAULT_TABLE table=test-table [id=521eb1cd39b94eaaaf4bd40332571ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:58:53.219627 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:53.219772 2722 tablet_bootstrap.cc:654] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:53.219797 2719 tablet_bootstrap.cc:654] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:53.220307 2722 tablet_bootstrap.cc:492] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3: No bootstrap required, opened a new log
I20260928 08:58:53.220368 2722 ts_tablet_manager.cc:1397] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:53.220422 2719 tablet_bootstrap.cc:492] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: No bootstrap required, opened a new log
I20260928 08:58:53.220466 2719 ts_tablet_manager.cc:1397] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 08:58:53.220476 2722 raft_consensus.cc:348] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.220557 2722 raft_consensus.cc:374] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:53.220583 2722 raft_consensus.cc:729] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Initialized, Role: FOLLOWER
I20260928 08:58:53.220609 2719 raft_consensus.cc:348] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.220652 2722 consensus_queue.cc:260] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.220700 2719 raft_consensus.cc:374] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:53.220736 2719 raft_consensus.cc:729] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f6550b2451441e9d70cbb6555babb7, State: Initialized, Role: FOLLOWER
I20260928 08:58:53.220746 2722 ts_tablet_manager.cc:1428] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:53.220779 2719 consensus_queue.cc:260] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.220876 2719 ts_tablet_manager.cc:1428] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:53.221060 2718 tablet_bootstrap.cc:492] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Bootstrap starting.
I20260928 08:58:53.221390 2718 tablet_bootstrap.cc:654] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:53.221959 2718 tablet_bootstrap.cc:492] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: No bootstrap required, opened a new log
I20260928 08:58:53.222066 2718 ts_tablet_manager.cc:1397] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:58:53.222190 2718 raft_consensus.cc:348] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.222252 2718 raft_consensus.cc:374] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:53.222291 2718 raft_consensus.cc:729] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084d19799cdf4f90b43f0553751bfcf1, State: Initialized, Role: FOLLOWER
I20260928 08:58:53.222342 2718 consensus_queue.cc:260] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.222443 2718 ts_tablet_manager.cc:1428] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 08:58:53.395370 2724 raft_consensus.cc:482] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:58:53.395519 2724 raft_consensus.cc:504] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.395730 2724 leader_election.cc:288] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643), 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:53.396189 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbb90edad2d48f8a22ef9cab6cd0dda" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7" is_pre_election: true
I20260928 08:58:53.396190 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbb90edad2d48f8a22ef9cab6cd0dda" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1" is_pre_election: true
I20260928 08:58:53.396308 2415 raft_consensus.cc:2454] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 0.
I20260928 08:58:53.396308 2554 raft_consensus.cc:2454] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 0.
I20260928 08:58:53.396519 2208 leader_election.cc:302] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6415149702a74917bbfc9eeb286128b3, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:53.396672 2724 raft_consensus.cc:2781] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:58:53.396728 2724 raft_consensus.cc:482] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:58:53.396761 2724 raft_consensus.cc:3037] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:53.397629 2724 raft_consensus.cc:504] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.397760 2724 leader_election.cc:288] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 election: Requested vote from peers 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643), 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549)
I20260928 08:58:53.397944 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbb90edad2d48f8a22ef9cab6cd0dda" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6550b2451441e9d70cbb6555babb7"
I20260928 08:58:53.397970 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbb90edad2d48f8a22ef9cab6cd0dda" candidate_uuid: "6415149702a74917bbfc9eeb286128b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d19799cdf4f90b43f0553751bfcf1"
I20260928 08:58:53.398028 2415 raft_consensus.cc:3037] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:53.398036 2554 raft_consensus.cc:3037] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:53.398764 2415 raft_consensus.cc:2454] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 1.
I20260928 08:58:53.398840 2554 raft_consensus.cc:2454] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6415149702a74917bbfc9eeb286128b3 in term 1.
I20260928 08:58:53.399032 2208 leader_election.cc:302] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6415149702a74917bbfc9eeb286128b3, 90f6550b2451441e9d70cbb6555babb7; no voters:
I20260928 08:58:53.399196 2724 raft_consensus.cc:2781] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:53.399266 2724 raft_consensus.cc:686] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6415149702a74917bbfc9eeb286128b3, State: Running, Role: LEADER
I20260928 08:58:53.399314 2724 consensus_queue.cc:237] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } }
I20260928 08:58:53.399889 2118 catalog_manager.cc:5495] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6415149702a74917bbfc9eeb286128b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6415149702a74917bbfc9eeb286128b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6415149702a74917bbfc9eeb286128b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 } health_report { overall_health: UNKNOWN } } }
I20260928 08:58:53.447294 2415 raft_consensus.cc:1261] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Refusing update from remote peer 6415149702a74917bbfc9eeb286128b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:53.447532 2554 raft_consensus.cc:1261] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Refusing update from remote peer 6415149702a74917bbfc9eeb286128b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:58:53.447538 2724 consensus_queue.cc:1035] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6550b2451441e9d70cbb6555babb7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:58:53.447788 2724 consensus_queue.cc:1035] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 6415149702a74917bbfc9eeb286128b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "084d19799cdf4f90b43f0553751bfcf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:58:53.451634 2690 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:58:54.968550 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:54.968654 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:58:54.969472 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b98194b8cc34de891fd7d563a66ec62: Sending DeleteTablet for 3 replicas of tablet 2cbb90edad2d48f8a22ef9cab6cd0dda
I20260928 08:58:54.969736 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260928 08:58:54.969771 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57536
I20260928 08:58:54.969780 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:58:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51118
I20260928 08:58:54.969960 2762 tablet_replica.cc:330] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: stopping tablet replica
I20260928 08:58:54.970023 2762 raft_consensus.cc:2229] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:54.970067 2762 raft_consensus.cc:2258] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:54.970158 2761 tablet_replica.cc:330] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: stopping tablet replica
I20260928 08:58:54.970148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 2190
I20260928 08:58:54.970216 2761 raft_consensus.cc:2229] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:58:54.971972 2761 raft_consensus.cc:2258] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:58:54.972810 2762 ts_tablet_manager.cc:1905] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:54.974038 2761 ts_tablet_manager.cc:1905] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:58:54.974267 2762 ts_tablet_manager.cc:1918] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:58:54.974323 2762 log.cc:1196] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/2cbb90edad2d48f8a22ef9cab6cd0dda
I20260928 08:58:54.974560 2762 ts_tablet_manager.cc:1939] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 084d19799cdf4f90b43f0553751bfcf1: Deleting consensus metadata
I20260928 08:58:54.974858 2104 catalog_manager.cc:4841] TS 084d19799cdf4f90b43f0553751bfcf1 (127.0.98.3:43549): tablet 2cbb90edad2d48f8a22ef9cab6cd0dda (table test-table [id=521eb1cd39b94eaaaf4bd40332571ba4]) successfully deleted
I20260928 08:58:54.975607 2761 ts_tablet_manager.cc:1918] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:58:54.975656 2761 log.cc:1196] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/2cbb90edad2d48f8a22ef9cab6cd0dda
I20260928 08:58:54.975884 2761 ts_tablet_manager.cc:1939] T 2cbb90edad2d48f8a22ef9cab6cd0dda P 90f6550b2451441e9d70cbb6555babb7: Deleting consensus metadata
I20260928 08:58:54.976202 2104 catalog_manager.cc:4841] TS 90f6550b2451441e9d70cbb6555babb7 (127.0.98.2:43643): tablet 2cbb90edad2d48f8a22ef9cab6cd0dda (table test-table [id=521eb1cd39b94eaaaf4bd40332571ba4]) successfully deleted
W20260928 08:58:54.976738 2103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41429 (error 108)
I20260928 08:58:54.976991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 2330
I20260928 08:58:54.977723 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:54.977110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41429 (local address 127.0.0.1:34538)
0928 08:58:54.977654 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:54.977693 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":56,"threads_started":1}
W20260928 08:58:54.977841 2103 catalog_manager.cc:4584] TS 6415149702a74917bbfc9eeb286128b3 (127.0.98.1:41429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cbb90edad2d48f8a22ef9cab6cd0dda: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41429: connect: Connection refused (error 111)
I20260928 08:58:54.983160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 2470
I20260928 08:58:54.988582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 2086
2026-09-28T08:58:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5890 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:55 dist-test-slave-286n krb5kdc[2767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:55 dist-test-slave-286n krb5kdc[2767](info): set up 2 sockets
Sep 28 08:58:55 dist-test-slave-286n krb5kdc[2767](info): commencing operation
krb5kdc: starting...
W20260928 08:58:56.024816 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:58:56.143613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:56.227506 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:56.227710 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:56.227731 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:56.228952 2780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:56.228991 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:56.229007 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:56.229019 2780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:56.229031 2780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:56.230482 2780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:56.230656 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:56.230818 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:56.235790 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.235781 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.235785 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:56.235814 2780 server_base.cc:1034] running on GCE node
I20260928 08:58:56.236306 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:56.236711 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:56.237903 2780 hybrid_clock.cc:648] HybridClock initialized: now 1790585936237882 us; error 32 us; skew 500 ppm
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:56.249796 2780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:56.250209 2780 webserver.cc:460] Webserver started at http://127.0.98.62:34577/ using document root <none> and password file <none>
I20260928 08:58:56.250414 2780 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:56.250456 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:56.250551 2780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:56.251575 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "fb5335a6b2f5a0ec2b6b6e67a98693d0"
server_key_iv: "63b6e378d60769981eda3c0436d81f7c"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.251864 2780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "fb5335a6b2f5a0ec2b6b6e67a98693d0"
server_key_iv: "63b6e378d60769981eda3c0436d81f7c"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.253213 2780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:56.253942 2795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.254148 2780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:56.254228 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "fb5335a6b2f5a0ec2b6b6e67a98693d0"
server_key_iv: "63b6e378d60769981eda3c0436d81f7c"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.254281 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:56.268412 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:56.270776 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:56.270928 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:56.274760 2780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41619
I20260928 08:58:56.275040 2847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41619 every 8 connection(s)
I20260928 08:58:56.275134 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:56.275887 2848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:56.278581 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de: Bootstrap starting.
I20260928 08:58:56.278808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 2780
I20260928 08:58:56.278895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:56.279093 392 external_mini_cluster.cc:1428] Setting key d1791f8c98df8ac60141444d83acb9fa
I20260928 08:58:56.279318 2848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:56.279673 2848 log.cc:824] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:56.280460 2848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de: No bootstrap required, opened a new log
I20260928 08:58:56.282691 2848 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260928 08:58:56.282857 2848 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:56.282887 2848 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d6b7de0c8b4c16a2cd88c3bf0333de, State: Initialized, Role: FOLLOWER
I20260928 08:58:56.283001 2848 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260928 08:58:56.283073 2848 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:56.283102 2848 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:56.283154 2848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:56.283813 2848 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260928 08:58:56.283924 2848 leader_election.cc:302] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5d6b7de0c8b4c16a2cd88c3bf0333de; no voters:
I20260928 08:58:56.284089 2848 leader_election.cc:288] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:56.284109 2853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:56.284214 2853 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [term 1 LEADER]: Becoming Leader. State: Replica: a5d6b7de0c8b4c16a2cd88c3bf0333de, State: Running, Role: LEADER
I20260928 08:58:56.284344 2848 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:56.284353 2853 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260928 08:58:56.284978 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } } }
I20260928 08:58:56.285070 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:56.285212 2854 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5d6b7de0c8b4c16a2cd88c3bf0333de. Latest consensus state: current_term: 1 leader_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d6b7de0c8b4c16a2cd88c3bf0333de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } } }
I20260928 08:58:56.285256 2854 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:56.285295 2862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:56.285857 2862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:56.289767 2862 catalog_manager.cc:1347] Generated new cluster ID: 73b847a3084749409cb0e916811a12ce
I20260928 08:58:56.289819 2862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:58:56.290781 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.279623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.98.62:41619)
0928 08:58:56.279797 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:56.279807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:56.279837 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:58:56.280236 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:56.280242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:56.280286 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:56.280402 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:56.281206 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.282102 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.283209 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.283820 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.283939 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.286910 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:56.286915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:56.286915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:56.286926 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:56.288779 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.289189 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.289192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.289192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.289282 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.289513 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.289514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.289514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.289823 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:56.289992 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:56.290433 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 08:58:56.290568 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260928 08:58:56.307440 2862 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:56.308214 2862 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:58:56.314925 2862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5d6b7de0c8b4c16a2cd88c3bf0333de: Generated new TSK 0
I20260928 08:58:56.315171 2862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:58:56.343720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41619
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:56.424499 2874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:56.424754 2874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:56.424782 2874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:56.425920 2874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:56.425959 2874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:56.425984 2874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:58:56.427429 2874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:56.427661 2874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:56.427896 2874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:56.432817 2880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:56.432914 2874 server_base.cc:1034] running on GCE node
W20260928 08:58:56.432837 2879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.432839 2882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:56.433260 2874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:56.433504 2874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:56.434661 2874 hybrid_clock.cc:648] HybridClock initialized: now 1790585936434658 us; error 42 us; skew 500 ppm
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:56.437235 2874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:58:56.437649 2874 webserver.cc:460] Webserver started at http://127.0.98.1:42525/ using document root <none> and password file <none>
I20260928 08:58:56.437858 2874 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:56.437923 2874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:56.438071 2874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:56.439137 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "3284d38f40f3437e99d67e13decf9de7"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "f39b9e5423a8f5025a90c013c82f9482"
server_key_iv: "a61cc9cf400be794213802f3c5b4a129"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.439489 2874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3284d38f40f3437e99d67e13decf9de7"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "f39b9e5423a8f5025a90c013c82f9482"
server_key_iv: "a61cc9cf400be794213802f3c5b4a129"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.440847 2874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:58:56.441629 2889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.441802 2874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:56.441958 2874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "3284d38f40f3437e99d67e13decf9de7"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "f39b9e5423a8f5025a90c013c82f9482"
server_key_iv: "a61cc9cf400be794213802f3c5b4a129"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.442034 2874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:56.464475 2874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:56.466338 2874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:56.466506 2874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:56.466745 2874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:56.467056 2874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:56.467106 2874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.467149 2874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:56.467178 2874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.472851 2874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33071
I20260928 08:58:56.472913 3002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33071 every 8 connection(s)
I20260928 08:58:56.473265 2874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:58:56.478935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 2874
I20260928 08:58:56.479071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:58:56.479260 392 external_mini_cluster.cc:1428] Setting key d9b1b47e0982df2870baea39e205bea8
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:56.482564 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.473969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34205 (local address 127.0.98.62:41619)
0928 08:58:56.474017 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:56.474019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:56.474214 (+ 195us) server_negotiation.cc:407] Connection header received
0928 08:58:56.475195 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:56.475196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:56.475206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:56.475266 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:56.475902 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.476395 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.477424 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.477618 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.477670 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.479738 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:56.479739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:56.479739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:56.479746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:56.481176 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.481589 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.481590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.481591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.481648 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.481900 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.481901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.481902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.482016 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:56.482104 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:56.482381 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 08:58:56.482464 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:58:56.482731 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.473926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41619 (local address 127.0.98.1:34205)
0928 08:58:56.474098 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:56.474140 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:56.475079 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:56.475334 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:56.475337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:56.475642 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:56.475828 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:56.475832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.476500 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.476500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:56.477351 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:56.477353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.477690 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.477691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:56.477789 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.478223 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:56.478231 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:56.479662 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:56.481273 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:56.481279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:56.481282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:56.481500 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:56.481718 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:56.481719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:56.481719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:56.481833 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:56.482094 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:56.482100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:56.482240 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 08:58:56.482349 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:58:56.482501 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260928 08:58:56.483187 3003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41619
I20260928 08:58:56.483276 3003 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:56.483460 3003 heartbeater.cc:511] Master 127.0.98.62:41619 requested a full tablet report, sending...
I20260928 08:58:56.483945 2812 ts_manager.cc:194] Registered new tserver with Master: 3284d38f40f3437e99d67e13decf9de7 (127.0.98.1:33071)
I20260928 08:58:56.484618 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34205
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:58:56.507798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41619
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:56.588050 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:56.588269 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:56.588295 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:56.589675 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:56.589723 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:56.589785 3008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:58:56.591487 3008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:56.591733 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:56.591969 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:56.596892 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.596903 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.596920 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:56.597164 3008 server_base.cc:1034] running on GCE node
I20260928 08:58:56.597431 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:56.597663 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:56.598834 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790585936598822 us; error 42 us; skew 500 ppm
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:56.601294 3008 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:58:56.601758 3008 webserver.cc:460] Webserver started at http://127.0.98.2:45907/ using document root <none> and password file <none>
I20260928 08:58:56.602000 3008 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:56.602069 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:56.602214 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:56.603230 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c9d22cf247f4635b449488fcdec77e1"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "c955e5b58f988623774f1a8f2610681a"
server_key_iv: "c6112b92753398e87c8533aa39a9073d"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.603603 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c9d22cf247f4635b449488fcdec77e1"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "c955e5b58f988623774f1a8f2610681a"
server_key_iv: "c6112b92753398e87c8533aa39a9073d"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.604928 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:58:56.605625 3023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.605815 3008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:56.605932 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "9c9d22cf247f4635b449488fcdec77e1"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "c955e5b58f988623774f1a8f2610681a"
server_key_iv: "c6112b92753398e87c8533aa39a9073d"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.606003 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:56.619182 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:56.621198 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:56.621359 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:56.621582 3008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:56.621878 3008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:56.621932 3008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.621971 3008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:56.621999 3008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.627861 3008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42355
I20260928 08:58:56.627911 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42355 every 8 connection(s)
I20260928 08:58:56.628259 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:58:56.632583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3008
I20260928 08:58:56.632706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:58:56.632907 392 external_mini_cluster.cc:1428] Setting key e37fcf9fa5b2ac095d6530a50c3a4230
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:56.638442 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.629059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34859 (local address 127.0.98.62:41619)
0928 08:58:56.629116 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:56.629151 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:56.629312 (+ 161us) server_negotiation.cc:407] Connection header received
0928 08:58:56.630215 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:56.630216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:56.630227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:56.630289 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:56.630905 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.631387 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.632500 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.632831 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.632909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.635424 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:56.635426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:56.635426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:56.635434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:56.636878 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.637286 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.637287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.637288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.637347 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.637620 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.637621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.637621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.637735 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:56.637820 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:56.638213 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 08:58:56.638315 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 08:58:56.638662 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.629000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41619 (local address 127.0.98.2:34859)
0928 08:58:56.629177 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:56.629220 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:56.630122 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:56.630356 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:56.630359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:56.630636 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:56.630830 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:56.630834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.631464 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.631465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:56.632384 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:56.632387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.632916 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.632917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:56.633076 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.633729 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:56.633738 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:56.635315 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:56.636969 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:56.636976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:56.636979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:56.637209 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:56.637415 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:56.637415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:56.637416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:56.637537 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:56.637850 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:56.637858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:56.638055 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 08:58:56.638187 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 08:58:56.638381 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260928 08:58:56.639237 3137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41619
I20260928 08:58:56.639324 3137 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:56.639519 3137 heartbeater.cc:511] Master 127.0.98.62:41619 requested a full tablet report, sending...
I20260928 08:58:56.639909 2812 ts_manager.cc:194] Registered new tserver with Master: 9c9d22cf247f4635b449488fcdec77e1 (127.0.98.2:42355)
I20260928 08:58:56.640442 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34859
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:58:56.660169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41619
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:56.746807 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:56.747007 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:56.747030 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:56.748109 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:56.748147 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:56.748172 3142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:58:56.749681 3142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:56.749881 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:56.750058 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:56.754818 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.754835 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:56.754905 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:56.754944 3142 server_base.cc:1034] running on GCE node
I20260928 08:58:56.755245 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:56.755470 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:56.756615 3142 hybrid_clock.cc:648] HybridClock initialized: now 1790585936756606 us; error 27 us; skew 500 ppm
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:56.759109 3142 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:58:56.759518 3142 webserver.cc:460] Webserver started at http://127.0.98.3:36783/ using document root <none> and password file <none>
I20260928 08:58:56.759750 3142 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:56.759814 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:56.759955 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:56.761030 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a97ea5fff7c48e580a64db7188d12bf"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "ecd993b96cef355cd9e6a86cb5588b55"
server_key_iv: "fc945cfabf0cd55b2bab73312880c1f3"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.761377 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a97ea5fff7c48e580a64db7188d12bf"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "ecd993b96cef355cd9e6a86cb5588b55"
server_key_iv: "fc945cfabf0cd55b2bab73312880c1f3"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.762647 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:56.763319 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.763484 3142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:56.763594 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "9a97ea5fff7c48e580a64db7188d12bf"
format_stamp: "Formatted at 2026-09-28 08:58:56 on dist-test-slave-286n"
server_key: "ecd993b96cef355cd9e6a86cb5588b55"
server_key_iv: "fc945cfabf0cd55b2bab73312880c1f3"
server_key_version: "encryptionkey@0"
I20260928 08:58:56.763662 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:56.776235 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:56.778468 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:56.778638 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:56.778878 3142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:56.779206 3142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:56.779260 3142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.779305 3142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:56.779340 3142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:56.785801 3142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32921
I20260928 08:58:56.785874 3270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32921 every 8 connection(s)
I20260928 08:58:56.786221 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[2767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:56.795063 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.786980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42429 (local address 127.0.98.62:41619)
0928 08:58:56.787031 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:56.787033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:56.787318 (+ 285us) server_negotiation.cc:407] Connection header received
0928 08:58:56.788132 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:56.788133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:56.788144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:56.788203 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:56.788843 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.789326 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.790262 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.790433 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.790508 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.792473 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:56.792474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:56.792475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:56.792483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:56.793732 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.794131 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.794132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.794133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.794192 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:56.794428 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:56.794428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:56.794429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:56.794542 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:56.794631 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:56.794892 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 08:58:56.794970 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 08:58:56.795274 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:56.786929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41619 (local address 127.0.98.3:42429)
0928 08:58:56.787084 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:56.787126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:56.788040 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:56.788255 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:56.788259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:56.788507 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:56.788765 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:56.788769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.789399 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.789400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:56.790194 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:56.790196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:56.790532 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:56.790534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:56.790661 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:56.791104 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:56.791112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:56.792378 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:56.793815 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:56.793821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:56.793824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:56.794037 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:56.794258 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:56.794259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:56.794259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:56.794362 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:56.794628 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:56.794639 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:56.794767 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 08:58:56.794872 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:58:56.795025 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260928 08:58:56.795320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3142
I20260928 08:58:56.795394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:58:56.795542 392 external_mini_cluster.cc:1428] Setting key c6f3b99346c51f76f3cc82469f72a17f
I20260928 08:58:56.795814 3271 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41619
I20260928 08:58:56.795917 3271 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:56.796104 3271 heartbeater.cc:511] Master 127.0.98.62:41619 requested a full tablet report, sending...
I20260928 08:58:56.796468 2812 ts_manager.cc:194] Registered new tserver with Master: 9a97ea5fff7c48e580a64db7188d12bf (127.0.98.3:32921)
I20260928 08:58:56.796908 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42429
I20260928 08:58:56.807266 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:58:56.818243 2851 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:58:56.817255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.0.98.62:41619)
0928 08:58:56.817344 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:56.817346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:56.817403 (+ 57us) server_negotiation.cc:407] Connection header received
0928 08:58:56.817450 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:56.817451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:56.817460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:56.817544 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:56.818160 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59504: BlockingRecv error: recv got EOF from 127.0.0.1:59504 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:58:56.819075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 2874
I20260928 08:58:56.823799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3008
I20260928 08:58:56.828176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3142
I20260928 08:58:56.832268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 2780
2026-09-28T08:58:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1841 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[3283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[3283](info): set up 2 sockets
Sep 28 08:58:56 dist-test-slave-286n krb5kdc[3283](info): commencing operation
krb5kdc: starting...
W20260928 08:58:57.916546 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:58:57 dist-test-slave-286n krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:58:58.035449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:58.119318 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:58.119560 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:58.119585 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:58.121031 3296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:58.121075 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:58.121090 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:58.121104 3296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:58.121115 3296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:58.122612 3296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:58.122820 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:58.122997 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:58.127997 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:58.128103 3296 server_base.cc:1034] running on GCE node
W20260928 08:58:58.128008 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:58.128014 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:58.128464 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:58.128793 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:58.129971 3296 hybrid_clock.cc:648] HybridClock initialized: now 1790585938129964 us; error 39 us; skew 500 ppm
Sep 28 08:58:58 dist-test-slave-286n krb5kdc[3283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:58.133538 3296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:58.134258 3296 webserver.cc:460] Webserver started at http://127.0.98.62:33017/ using document root <none> and password file <none>
I20260928 08:58:58.134474 3296 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:58.134526 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:58.134631 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:58.135820 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "8f3f25d5a5f94ef8b98d13300f04e059"
format_stamp: "Formatted at 2026-09-28 08:58:58 on dist-test-slave-286n"
server_key: "d7449c8e1f160193de5aee759c2cf654"
server_key_iv: "ad88cfa16868dcfa793fa3ecd3e3e153"
server_key_version: "encryptionkey@0"
I20260928 08:58:58.136157 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f3f25d5a5f94ef8b98d13300f04e059"
format_stamp: "Formatted at 2026-09-28 08:58:58 on dist-test-slave-286n"
server_key: "d7449c8e1f160193de5aee759c2cf654"
server_key_iv: "ad88cfa16868dcfa793fa3ecd3e3e153"
server_key_version: "encryptionkey@0"
I20260928 08:58:58.137943 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:58:58.138850 3311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:58.139048 3296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:58.139129 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "8f3f25d5a5f94ef8b98d13300f04e059"
format_stamp: "Formatted at 2026-09-28 08:58:58 on dist-test-slave-286n"
server_key: "d7449c8e1f160193de5aee759c2cf654"
server_key_iv: "ad88cfa16868dcfa793fa3ecd3e3e153"
server_key_version: "encryptionkey@0"
I20260928 08:58:58.139186 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:58.149461 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:58.151821 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:58.151935 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:58.155862 3296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35705
I20260928 08:58:58.155893 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35705 every 8 connection(s)
I20260928 08:58:58.156260 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:58.157178 3364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:58.159557 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059: Bootstrap starting.
I20260928 08:58:58.160171 3364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:58.160487 3364 log.cc:824] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:58.160488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3296
I20260928 08:58:58.160624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:58.160818 392 external_mini_cluster.cc:1428] Setting key fd6eb6a4353c2bb9f470c45fb606dc7e
I20260928 08:58:58.161324 3364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059: No bootstrap required, opened a new log
I20260928 08:58:58.163465 3364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260928 08:58:58.163609 3364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:58.163630 3364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3f25d5a5f94ef8b98d13300f04e059, State: Initialized, Role: FOLLOWER
I20260928 08:58:58.163760 3364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260928 08:58:58.163836 3364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:58.163890 3364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:58.163961 3364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:58.164536 3364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260928 08:58:58.164686 3364 leader_election.cc:302] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f3f25d5a5f94ef8b98d13300f04e059; no voters:
I20260928 08:58:58.164870 3364 leader_election.cc:288] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:58.164899 3369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:58.165010 3369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [term 1 LEADER]: Becoming Leader. State: Replica: 8f3f25d5a5f94ef8b98d13300f04e059, State: Running, Role: LEADER
I20260928 08:58:58.165108 3369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260928 08:58:58.165138 3364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:58.165470 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } } }
I20260928 08:58:58.165628 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:58.165465 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3f25d5a5f94ef8b98d13300f04e059. Latest consensus state: current_term: 1 leader_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f25d5a5f94ef8b98d13300f04e059" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } } }
I20260928 08:58:58.165791 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:58.165968 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:58.166561 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:58.168228 3376 catalog_manager.cc:1347] Generated new cluster ID: 814651dd4c0547e98226d33133a999e9
I20260928 08:58:58.168292 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:58:58 dist-test-slave-286n krb5kdc[3283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:58.175702 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:58.161353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.0.98.62:35705)
0928 08:58:58.161511 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:58.161519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:58.161548 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:58:58.163473 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:58.163482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:58.163542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:58.163835 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:58.164683 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:58.166086 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:58.167246 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:58.167859 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:58.167964 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:58.171000 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:58.171005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:58.171005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:58.171019 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:58.173815 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:58.174264 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:58.174268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:58.174268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:58.174388 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:58.174615 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:58.174615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:58.174616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:58.174923 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:58.175085 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:58.175199 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 08:58:58.175461 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260928 08:58:58.176708 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:58:58.183864 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:58.184723 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:58:58.192915 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f3f25d5a5f94ef8b98d13300f04e059: Generated new TSK 0
I20260928 08:58:58.193132 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:58:58.193693 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:58.189038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.0.98.62:35705)
0928 08:58:58.189098 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:58.189100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:58.189234 (+ 134us) server_negotiation.cc:407] Connection header received
0928 08:58:58.189294 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:58.189295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:58.189306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:58.189384 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:58.189826 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:58.190629 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:58.191742 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:58.192092 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:58.192160 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:58.193105 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:58.193106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:58.193107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:58:58.193321 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:58.193339 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:58:58.193346 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:58.193380 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:58.193551 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 08:58:58.193582 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":34}
I20260928 08:58:58.198755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3296
2026-09-28T08:58:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1366 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:58:58 dist-test-slave-286n krb5kdc[3397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:58:58 dist-test-slave-286n krb5kdc[3397](info): set up 2 sockets
Sep 28 08:58:58 dist-test-slave-286n krb5kdc[3397](info): commencing operation
krb5kdc: starting...
W20260928 08:58:59.264621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:58:59.387378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:59.467361 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:59.467639 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:59.467682 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:59.468909 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:58:59.468982 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:59.469007 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:59.469029 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:58:59.469071 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:58:59.470628 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:59.470912 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:59.471138 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:59.476114 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.476143 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:59.476179 3410 server_base.cc:1034] running on GCE node
W20260928 08:58:59.476127 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:59.476624 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:59.476976 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:59.478143 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790585939478134 us; error 42 us; skew 500 ppm
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:59.480793 3410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:58:59.481216 3410 webserver.cc:460] Webserver started at http://127.0.98.62:40735/ using document root <none> and password file <none>
I20260928 08:58:59.481458 3410 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:59.481535 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:59.481710 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:59.482815 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "4d174e7330ff4625b5f6e72c1cf335c0"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "d4bc89c73622e806d8bf175d1d4d9d60"
server_key_iv: "dde12904bdd7c899e4c15d95ce9dcabd"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.483179 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d174e7330ff4625b5f6e72c1cf335c0"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "d4bc89c73622e806d8bf175d1d4d9d60"
server_key_iv: "dde12904bdd7c899e4c15d95ce9dcabd"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.484459 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:58:59.485320 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.485487 3410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:59.485594 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "4d174e7330ff4625b5f6e72c1cf335c0"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "d4bc89c73622e806d8bf175d1d4d9d60"
server_key_iv: "dde12904bdd7c899e4c15d95ce9dcabd"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.485679 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:59.506315 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:59.508177 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:59.508349 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:59.512275 3410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34067
I20260928 08:58:59.512332 3477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34067 every 8 connection(s)
I20260928 08:58:59.512722 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:58:59.513758 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:58:59.516095 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0: Bootstrap starting.
I20260928 08:58:59.516714 3478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0: Neither blocks nor log segments found. Creating new log.
I20260928 08:58:59.517000 3478 log.cc:824] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:58:59.517696 3478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0: No bootstrap required, opened a new log
I20260928 08:58:59.519263 3478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260928 08:58:59.519433 3478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:58:59.519460 3478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d174e7330ff4625b5f6e72c1cf335c0, State: Initialized, Role: FOLLOWER
I20260928 08:58:59.519652 3478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260928 08:58:59.519733 3478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:58:59.519757 3478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:58:59.519825 3478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:58:59.520627 3478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260928 08:58:59.520797 3478 leader_election.cc:302] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d174e7330ff4625b5f6e72c1cf335c0; no voters:
I20260928 08:58:59.520977 3478 leader_election.cc:288] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:58:59.521011 3481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:58:59.521227 3481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d174e7330ff4625b5f6e72c1cf335c0, State: Running, Role: LEADER
I20260928 08:58:59.521247 3478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:58:59.521318 3481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260928 08:58:59.521696 3482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } } }
I20260928 08:58:59.521698 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d174e7330ff4625b5f6e72c1cf335c0. Latest consensus state: current_term: 1 leader_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d174e7330ff4625b5f6e72c1cf335c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } } }
I20260928 08:58:59.521869 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:59.521844 3482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:58:59.522105 3490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:58:59.522358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3410
I20260928 08:58:59.522459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:58:59.522662 392 external_mini_cluster.cc:1428] Setting key fe96a3ed1c08c22cf2953d773767b74a
I20260928 08:58:59.522866 3490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:58:59.524462 3490 catalog_manager.cc:1347] Generated new cluster ID: 30e97445114e4216baf357eca9ccb53a
I20260928 08:58:59.524514 3490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:58:59.530440 3490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:58:59.531126 3490 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:59.535925 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:59.523160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.0.98.62:34067)
0928 08:58:59.523575 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:59.523595 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:59.524859 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 08:58:59.524990 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:59.524998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:59.525053 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:59.525207 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:59.525928 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.527369 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.528423 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.529101 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.529206 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:59.532259 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:59.532263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:59.532263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:59.532276 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:59.534278 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:59.534681 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:59.534684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:59.534685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:59.534792 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:59.535011 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:59.535012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:59.535012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:59.535270 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:59.535403 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:59.535555 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 08:58:59.535693 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":82,"threads_started":1}
I20260928 08:58:59.550061 3490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d174e7330ff4625b5f6e72c1cf335c0: Generated new TSK 0
I20260928 08:58:59.550406 3490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:58:59.584486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:59.666290 3504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:59.666524 3504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:59.666560 3504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:59.667695 3504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:59.667755 3504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:59.667800 3504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:58:59.669399 3504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:59.669646 3504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:59.669873 3504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:59.675011 3512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.675026 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.675021 3509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:59.675037 3504 server_base.cc:1034] running on GCE node
I20260928 08:58:59.675384 3504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:59.675616 3504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:59.676719 3504 hybrid_clock.cc:648] HybridClock initialized: now 1790585939676709 us; error 33 us; skew 500 ppm
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:59.679265 3504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:58:59.679661 3504 webserver.cc:460] Webserver started at http://127.0.98.1:45191/ using document root <none> and password file <none>
I20260928 08:58:59.679836 3504 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:59.679873 3504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:59.679968 3504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:59.681016 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "8bbc37cde9254b1189037610442bde4f"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "cd14b1a8a47c9b0e21bba451cf4c0bbb"
server_key_iv: "a8d2699fb8a922c4fd81f010aa9267b9"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.681303 3504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8bbc37cde9254b1189037610442bde4f"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "cd14b1a8a47c9b0e21bba451cf4c0bbb"
server_key_iv: "a8d2699fb8a922c4fd81f010aa9267b9"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.682588 3504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:58:59.683320 3519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.683570 3504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:58:59.683647 3504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "8bbc37cde9254b1189037610442bde4f"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "cd14b1a8a47c9b0e21bba451cf4c0bbb"
server_key_iv: "a8d2699fb8a922c4fd81f010aa9267b9"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.683738 3504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:59.704772 3504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:59.707115 3504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:59.707252 3504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:59.707516 3504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:59.707844 3504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:59.707876 3504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.707898 3504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:59.707942 3504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.713501 3504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33903
I20260928 08:58:59.713599 3632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33903 every 8 connection(s)
I20260928 08:58:59.713971 3504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:58:59.719887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3504
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:59.719974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:58:59.720173 392 external_mini_cluster.cc:1428] Setting key e73e9b828e56b1240b918e7be5662191
I20260928 08:58:59.723409 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:59.714764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37163 (local address 127.0.98.62:34067)
0928 08:58:59.714839 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:59.714842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:59.715053 (+ 211us) server_negotiation.cc:407] Connection header received
0928 08:58:59.715562 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:59.715563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:59.715573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:59.715630 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:59.716563 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.717101 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.718082 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.718294 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.718347 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:59.720431 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:59.720432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:59.720433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:59.720440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:59.721961 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:59.722379 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:59.722380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:59.722381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:59.722455 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:59.722722 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:59.722723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:59.722724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:59.722858 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:59.722952 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:59.723237 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 08:58:59.723324 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:58:59.723598 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:59.714728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.1:37163)
0928 08:58:59.714933 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:59.714969 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:59.715454 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:59.715698 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:59.715702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:59.716274 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:59.716457 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:59.716462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.717195 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.717196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:59.717989 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:59.717991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.718364 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.718365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:59.718461 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:59.718908 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:59.718916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:59.720341 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:59.722049 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:59.722056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:59.722058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:59.722310 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:59.722531 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:59.722532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:59.722533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:59.722644 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:59.722957 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:59.722963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:59.723097 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 08:58:59.723202 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:58:59.723353 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260928 08:58:59.724040 3633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34067
I20260928 08:58:59.724145 3633 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:59.724363 3633 heartbeater.cc:511] Master 127.0.98.62:34067 requested a full tablet report, sending...
I20260928 08:58:59.724933 3442 ts_manager.cc:194] Registered new tserver with Master: 8bbc37cde9254b1189037610442bde4f (127.0.98.1:33903)
I20260928 08:58:59.725922 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37163
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:58:59.745752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:59.828140 3638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:59.828351 3638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:59.828374 3638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:59.829586 3638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:59.829627 3638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:59.829680 3638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:58:59.831323 3638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:59.831558 3638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:59.831779 3638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:58:59.836690 3644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.836697 3643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.836697 3646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:59.836934 3638 server_base.cc:1034] running on GCE node
I20260928 08:58:59.837131 3638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:59.837369 3638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:59.838531 3638 hybrid_clock.cc:648] HybridClock initialized: now 1790585939838534 us; error 44 us; skew 500 ppm
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:59.841102 3638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:58:59.841570 3638 webserver.cc:460] Webserver started at http://127.0.98.2:46675/ using document root <none> and password file <none>
I20260928 08:58:59.841804 3638 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:59.841876 3638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:59.842024 3638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:59.843091 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "7865bad6fcbd42628b4c15b233e439ff"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "f423a99ff17d152d96e9d3ceabac2847"
server_key_iv: "db595a68df7c5446aa2962ed3f656ddc"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.843472 3638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7865bad6fcbd42628b4c15b233e439ff"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "f423a99ff17d152d96e9d3ceabac2847"
server_key_iv: "db595a68df7c5446aa2962ed3f656ddc"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.844935 3638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:58:59.845672 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.845881 3638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:58:59.845978 3638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "7865bad6fcbd42628b4c15b233e439ff"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "f423a99ff17d152d96e9d3ceabac2847"
server_key_iv: "db595a68df7c5446aa2962ed3f656ddc"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.846055 3638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:58:59.859251 3638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:58:59.861379 3638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:58:59.861541 3638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:58:59.861769 3638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:58:59.862080 3638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:58:59.862133 3638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.862174 3638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:58:59.862201 3638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.868211 3638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33255
I20260928 08:58:59.868257 3766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33255 every 8 connection(s)
I20260928 08:58:59.868568 3638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:58:59.871014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3638
I20260928 08:58:59.871130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:58:59.871323 392 external_mini_cluster.cc:1428] Setting key de0983b5db573f07bcc3f9e48186026d
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:58:59.878944 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:59.869340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49867 (local address 127.0.98.62:34067)
0928 08:58:59.869410 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:58:59.869412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:58:59.869641 (+ 229us) server_negotiation.cc:407] Connection header received
0928 08:58:59.870090 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:58:59.870091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:58:59.870102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:58:59.870165 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:58:59.871200 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.872176 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.873383 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.873570 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.873630 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:59.876068 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:58:59.876069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:58:59.876070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:58:59.876078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:58:59.877565 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:59.877990 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:59.877991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:59.877991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:59.878053 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:58:59.878301 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:58:59.878302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:58:59.878302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:58:59.878423 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:58:59.878500 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:58:59.878762 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 08:58:59.878846 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 08:58:59.879154 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:58:59.869292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.2:49867)
0928 08:58:59.869485 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:58:59.869528 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:58:59.870008 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:58:59.870232 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:58:59.870235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:58:59.870809 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:58:59.871037 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:59.871043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.872251 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.872253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:59.873282 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:58:59.873286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:58:59.873653 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:58:59.873653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:58:59.873800 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:58:59.874339 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:58:59.874349 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:58:59.875971 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:58:59.877665 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:59.877671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:59.877673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:59.877903 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:59.878117 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:58:59.878118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:58:59.878118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:58:59.878225 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:58:59.878491 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:58:59.878497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:58:59.878635 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 08:58:59.878743 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:58:59.878911 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260928 08:58:59.879592 3767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34067
I20260928 08:58:59.879688 3767 heartbeater.cc:464] Registering TS with master...
I20260928 08:58:59.879829 3767 heartbeater.cc:511] Master 127.0.98.62:34067 requested a full tablet report, sending...
I20260928 08:58:59.880220 3442 ts_manager.cc:194] Registered new tserver with Master: 7865bad6fcbd42628b4c15b233e439ff (127.0.98.2:33255)
I20260928 08:58:59.880574 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:58:59.896201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:58:59.977109 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:58:59.977340 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:58:59.977377 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:58:59.978579 3772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:58:59.978652 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:58:59.978699 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:58:59.980178 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:58:59.980445 3772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:58:59.980734 3772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:58:59.985879 3772 server_base.cc:1034] running on GCE node
W20260928 08:58:59.985848 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.985862 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:58:59.985859 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:58:59.986284 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:58:59.986564 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:58:59.987740 3772 hybrid_clock.cc:648] HybridClock initialized: now 1790585939987719 us; error 39 us; skew 500 ppm
Sep 28 08:58:59 dist-test-slave-286n krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:58:59.990279 3772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:58:59.990741 3772 webserver.cc:460] Webserver started at http://127.0.98.3:39649/ using document root <none> and password file <none>
I20260928 08:58:59.990986 3772 fs_manager.cc:359] Metadata directory not provided
I20260928 08:58:59.991055 3772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:58:59.991199 3772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:58:59.992267 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9ba95035cd547139c62e8ce8b35523e"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "2b6113c420b6c84e6252745639655416"
server_key_iv: "429f50f4abacfe61e6dc315fe24ae1d0"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.992677 3772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9ba95035cd547139c62e8ce8b35523e"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "2b6113c420b6c84e6252745639655416"
server_key_iv: "429f50f4abacfe61e6dc315fe24ae1d0"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.994082 3772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:58:59.994880 3787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:58:59.995074 3772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:58:59.995167 3772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "c9ba95035cd547139c62e8ce8b35523e"
format_stamp: "Formatted at 2026-09-28 08:58:59 on dist-test-slave-286n"
server_key: "2b6113c420b6c84e6252745639655416"
server_key_iv: "429f50f4abacfe61e6dc315fe24ae1d0"
server_key_version: "encryptionkey@0"
I20260928 08:58:59.995221 3772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:00.013154 3772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:00.015432 3772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:00.015564 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:00.015759 3772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:00.016029 3772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:00.016060 3772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:00.016083 3772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:00.016096 3772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:00.022231 3772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44813
I20260928 08:59:00.022289 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44813 every 8 connection(s)
I20260928 08:59:00.022625 3772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:59:00 dist-test-slave-286n krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:00.031286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3772
I20260928 08:59:00.031383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:00.031657 392 external_mini_cluster.cc:1428] Setting key 014b39ee0a9ce26448785e7c134f7e3c
I20260928 08:59:00.033066 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:00.023316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48023 (local address 127.0.98.62:34067)
0928 08:59:00.023389 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:00.023391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:00.023737 (+ 346us) server_negotiation.cc:407] Connection header received
0928 08:59:00.024241 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:00.024243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:00.024254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:00.024716 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:00.025738 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:00.026251 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:00.027295 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:00.027487 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:00.027549 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:00.030057 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:00.030058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:00.030059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:00.030066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:00.031441 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:00.031947 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:00.031949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:00.031949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:00.032017 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:00.032344 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:00.032345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:00.032345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:00.032487 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:00.032574 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:00.032913 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 08:59:00.033001 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 08:59:00.033192 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:00.023282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.3:48023)
0928 08:59:00.023583 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:00.023618 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:00.024086 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:00.024799 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:00.024803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:00.025418 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:00.025640 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:00.025645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:00.026338 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:00.026339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:00.027212 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:00.027215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:00.027590 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:00.027591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:00.027724 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:00.028337 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:00.028347 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:00.029960 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:00.031537 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:00.031551 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:00.031554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:00.031850 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:00.032097 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:00.032098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:00.032098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:00.032247 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:00.032576 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:00.032582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:00.032766 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 08:59:00.032860 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:59:00.032992 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":81,"threads_started":1}
I20260928 08:59:00.033787 3901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34067
I20260928 08:59:00.033906 3901 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:00.034101 3901 heartbeater.cc:511] Master 127.0.98.62:34067 requested a full tablet report, sending...
I20260928 08:59:00.034489 3442 ts_manager.cc:194] Registered new tserver with Master: c9ba95035cd547139c62e8ce8b35523e (127.0.98.3:44813)
I20260928 08:59:00.034904 3442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48023
I20260928 08:59:00.043744 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:59:00 dist-test-slave-286n krb5kdc[3397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 08:59:00 dist-test-slave-286n krb5kdc[3397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585940, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:00.066386 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:00.059204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.0.98.62:34067)
0928 08:59:00.059292 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:00.059294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:00.059398 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:59:00.059506 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:00.059507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:00.059520 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:00.059578 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:00.060133 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:00.060681 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:00.061441 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:00.061649 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:00.061735 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:00.064118 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:00.064119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:00.064119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:00.064126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:00.065429 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:00.065726 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:00.065727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:00.065727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:00.065777 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:00.065929 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:00.065929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:00.065930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:00.066081 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:00.066155 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:00.066236 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:59:00.066293 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 08:59:00.066655 3442 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41466
I20260928 08:59:00.067094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3504
I20260928 08:59:00.072057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3638
I20260928 08:59:00.076249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3772
I20260928 08:59:00.080538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3410
2026-09-28T08:59:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1882 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:00 dist-test-slave-286n krb5kdc[3913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:00 dist-test-slave-286n krb5kdc[3913](info): set up 2 sockets
Sep 28 08:59:00 dist-test-slave-286n krb5kdc[3913](info): commencing operation
krb5kdc: starting...
W20260928 08:59:01.124913 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:59:01.243490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:01.323060 3926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:01.323318 3926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:01.323355 3926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:01.324560 3926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:01.324627 3926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:01.324671 3926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:01.324693 3926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:01.324747 3926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:01.326210 3926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:01.326480 3926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:01.326705 3926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:01.331791 3934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.331779 3931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.331782 3932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:01.331895 3926 server_base.cc:1034] running on GCE node
I20260928 08:59:01.332319 3926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:01.332675 3926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:01.333850 3926 hybrid_clock.cc:648] HybridClock initialized: now 1790585941333822 us; error 46 us; skew 500 ppm
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:01.342720 3926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:59:01.343202 3926 webserver.cc:460] Webserver started at http://127.0.98.62:38859/ using document root <none> and password file <none>
I20260928 08:59:01.343433 3926 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:01.343482 3926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:01.343640 3926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:01.344762 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "6916e38582564f469eb31e4527607253"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "e444be2bc4030d348ae8307d42e94fe9"
server_key_iv: "1ebd35c5d7421032636f4844ad5c0a22"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.345122 3926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "6916e38582564f469eb31e4527607253"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "e444be2bc4030d348ae8307d42e94fe9"
server_key_iv: "1ebd35c5d7421032636f4844ad5c0a22"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.346442 3926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:59:01.347175 3941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.347347 3926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.347453 3926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "6916e38582564f469eb31e4527607253"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "e444be2bc4030d348ae8307d42e94fe9"
server_key_iv: "1ebd35c5d7421032636f4844ad5c0a22"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.347527 3926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:01.357019 3926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:01.359357 3926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:01.359515 3926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:01.363323 3926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33489
I20260928 08:59:01.363360 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33489 every 8 connection(s)
I20260928 08:59:01.363721 3926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:01.364600 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:01.366814 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253: Bootstrap starting.
I20260928 08:59:01.367312 3994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:01.367550 3994 log.cc:824] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:01.368139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 3926
I20260928 08:59:01.368158 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253: No bootstrap required, opened a new log
I20260928 08:59:01.368265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:01.368459 392 external_mini_cluster.cc:1428] Setting key ce6e9401ee29271ea0c21a5768c365c3
I20260928 08:59:01.370776 3994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6916e38582564f469eb31e4527607253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33489 } }
I20260928 08:59:01.370949 3994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:01.370985 3994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6916e38582564f469eb31e4527607253, State: Initialized, Role: FOLLOWER
I20260928 08:59:01.371169 3994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6916e38582564f469eb31e4527607253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33489 } }
I20260928 08:59:01.371253 3994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:01.371275 3994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:01.371336 3994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:01.371960 3994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6916e38582564f469eb31e4527607253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33489 } }
I20260928 08:59:01.372064 3994 leader_election.cc:302] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6916e38582564f469eb31e4527607253; no voters:
I20260928 08:59:01.372223 3994 leader_election.cc:288] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:01.372300 3999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:01.372457 3999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [term 1 LEADER]: Becoming Leader. State: Replica: 6916e38582564f469eb31e4527607253, State: Running, Role: LEADER
I20260928 08:59:01.372457 3994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:01.373232 3999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6916e38582564f469eb31e4527607253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33489 } }
I20260928 08:59:01.373998 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6916e38582564f469eb31e4527607253. Latest consensus state: current_term: 1 leader_uuid: "6916e38582564f469eb31e4527607253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6916e38582564f469eb31e4527607253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33489 } } }
I20260928 08:59:01.374068 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:01.373836 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6916e38582564f469eb31e4527607253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6916e38582564f469eb31e4527607253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33489 } } }
I20260928 08:59:01.374249 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253 [sys.catalog]: This master's current role is: LEADER
W20260928 08:59:01.374545 4013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:59:01.374600 4013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:59:01.374691 4014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:01.374886 4014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:01.376509 4014 catalog_manager.cc:1347] Generated new cluster ID: 258ff77e7573469199b718155d185451
I20260928 08:59:01.376567 4014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:01.380770 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.369038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.0.98.62:33489)
0928 08:59:01.369242 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:01.369252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:01.369284 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:59:01.370729 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:01.370737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:01.370800 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:01.370965 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:01.371712 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.373033 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.374233 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.374831 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.374933 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.377580 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:01.377584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:01.377585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:01.377596 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:01.379255 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.379620 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.379623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.379623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.379707 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.379914 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.379914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.379915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.380140 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:01.380312 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:01.380387 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:59:01.380533 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260928 08:59:01.391352 4014 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:01.391906 4014 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:01.397274 4014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6916e38582564f469eb31e4527607253: Generated new TSK 0
I20260928 08:59:01.397454 4014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:59:01.429591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33489
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:01.513331 4020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:01.513528 4020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:01.513551 4020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:01.514685 4020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:01.514724 4020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:01.514750 4020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:01.516156 4020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:01.516332 4020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:01.516497 4020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:01.521220 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.521217 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.521340 4028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:01.521575 4020 server_base.cc:1034] running on GCE node
I20260928 08:59:01.521799 4020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:01.522064 4020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:01.523214 4020 hybrid_clock.cc:648] HybridClock initialized: now 1790585941523204 us; error 30 us; skew 500 ppm
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:01.525924 4020 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:59:01.526314 4020 webserver.cc:460] Webserver started at http://127.0.98.1:39969/ using document root <none> and password file <none>
I20260928 08:59:01.526505 4020 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:01.526546 4020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:01.526631 4020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:01.527634 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "b353d18f989b45a888920eab5a6eb829"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "9afb81a22a72643bf529a759f04d79a0"
server_key_iv: "58f4962bf38c7bbf4d80e347e4d077cd"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.528013 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b353d18f989b45a888920eab5a6eb829"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "9afb81a22a72643bf529a759f04d79a0"
server_key_iv: "58f4962bf38c7bbf4d80e347e4d077cd"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.529436 4020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:01.530267 4035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.530504 4020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:01.530601 4020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "b353d18f989b45a888920eab5a6eb829"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "9afb81a22a72643bf529a759f04d79a0"
server_key_iv: "58f4962bf38c7bbf4d80e347e4d077cd"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.530659 4020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:01.547540 4020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:01.549736 4020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:01.549860 4020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:01.550065 4020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:01.550391 4020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:01.550433 4020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.550458 4020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:01.550473 4020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.556766 4020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41385
I20260928 08:59:01.556861 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41385 every 8 connection(s)
I20260928 08:59:01.557195 4020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:01.565383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4020
I20260928 08:59:01.565514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:01.565716 392 external_mini_cluster.cc:1428] Setting key b0d1ab8800584e11df038d73da67538a
I20260928 08:59:01.567632 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.557947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33489 (local address 127.0.98.1:48413)
0928 08:59:01.558141 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:01.558177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:01.559180 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:01.559492 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:01.559496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:01.559757 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:01.559950 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:01.559955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.560750 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.560751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:01.561575 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:01.561577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.562010 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.562011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:01.562138 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.562679 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:01.562688 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:01.564126 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:01.566165 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:01.566171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:01.566174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:01.566508 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:01.566735 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:01.566736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:01.566736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:01.566833 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:01.567132 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:01.567136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:01.567250 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 08:59:01.567335 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:59:01.567452 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260928 08:59:01.567674 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.557970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48413 (local address 127.0.98.62:33489)
0928 08:59:01.558036 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:01.558038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:01.558274 (+ 236us) server_negotiation.cc:407] Connection header received
0928 08:59:01.559307 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:01.559308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:01.559323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:01.559405 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:01.560057 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.560593 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.561697 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.561905 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.562001 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.564220 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:01.564222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:01.564222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:01.564229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:01.566068 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.566591 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.566592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.566593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.566662 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.566898 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.566899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.566899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.567061 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:01.567141 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:01.567403 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 08:59:01.567506 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 08:59:01.568495 4149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33489
I20260928 08:59:01.568606 4149 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:01.568823 4149 heartbeater.cc:511] Master 127.0.98.62:33489 requested a full tablet report, sending...
I20260928 08:59:01.569377 3958 ts_manager.cc:194] Registered new tserver with Master: b353d18f989b45a888920eab5a6eb829 (127.0.98.1:41385)
I20260928 08:59:01.570045 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48413
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:59:01.595191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33489
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:01.676093 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:01.676317 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:01.676357 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:01.677582 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:01.677659 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:01.677707 4154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:01.679207 4154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:01.679463 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:01.679720 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:01.684674 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.684613 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:01.684877 4154 server_base.cc:1034] running on GCE node
W20260928 08:59:01.684613 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:01.685122 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:01.685365 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:01.686528 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790585941686523 us; error 34 us; skew 500 ppm
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:01.688905 4154 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:59:01.689375 4154 webserver.cc:460] Webserver started at http://127.0.98.2:37249/ using document root <none> and password file <none>
I20260928 08:59:01.689636 4154 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:01.689704 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:01.689853 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:01.690876 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "c412bb090ec0435888a42b0d7e85abe2"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "ab17f4f3714632dd4049c6c66f3c72df"
server_key_iv: "9f46cf139073129256750dc64a0270ab"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.691246 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c412bb090ec0435888a42b0d7e85abe2"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "ab17f4f3714632dd4049c6c66f3c72df"
server_key_iv: "9f46cf139073129256750dc64a0270ab"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.692535 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 08:59:01.693316 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.693483 4154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:01.693580 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "c412bb090ec0435888a42b0d7e85abe2"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "ab17f4f3714632dd4049c6c66f3c72df"
server_key_iv: "9f46cf139073129256750dc64a0270ab"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.693661 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:01.706333 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:01.708153 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:01.708307 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:01.708549 4154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:01.708889 4154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:01.708942 4154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.708992 4154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:01.709021 4154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.716261 4154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46387
I20260928 08:59:01.716331 4282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46387 every 8 connection(s)
I20260928 08:59:01.716717 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:59:01.720068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4154
I20260928 08:59:01.720186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:01.720347 392 external_mini_cluster.cc:1428] Setting key 813dded95b6c18f76a63ecec451658f5
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:01.727456 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.717615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33667 (local address 127.0.98.62:33489)
0928 08:59:01.717662 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:01.717696 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:01.718500 (+ 804us) server_negotiation.cc:407] Connection header received
0928 08:59:01.719641 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:01.719643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:01.719657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:01.719766 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:01.720619 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.721211 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.722223 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.722413 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.722485 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.724549 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:01.724550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:01.724551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:01.724558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:01.726099 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.726496 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.726497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.726498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.726599 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.726831 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.726832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.726832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.726944 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:01.727019 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:01.727284 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 08:59:01.727363 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:59:01.727664 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.717606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33489 (local address 127.0.98.2:33667)
0928 08:59:01.718352 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:01.718397 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:01.719516 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:01.719905 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:01.719909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:01.720274 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:01.720464 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:01.720470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.721315 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.721316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:01.722123 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:01.722125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.722489 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.722490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:01.722614 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.723141 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:01.723149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:01.724457 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:01.726181 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:01.726187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:01.726190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:01.726407 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:01.726656 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:01.726656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:01.726657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:01.726755 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:01.727009 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:01.727014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:01.727153 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 08:59:01.727264 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 08:59:01.727419 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 08:59:01.728340 4283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33489
I20260928 08:59:01.728446 4283 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:01.728695 4283 heartbeater.cc:511] Master 127.0.98.62:33489 requested a full tablet report, sending...
I20260928 08:59:01.729188 3958 ts_manager.cc:194] Registered new tserver with Master: c412bb090ec0435888a42b0d7e85abe2 (127.0.98.2:46387)
I20260928 08:59:01.729573 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:59:01.747886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33489
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:01.831727 4288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:01.831944 4288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:01.831967 4288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:01.833237 4288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:01.833283 4288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:01.833312 4288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:01.834728 4288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:01.834996 4288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:01.835206 4288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:01.840308 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.840267 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:01.840266 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:01.840487 4288 server_base.cc:1034] running on GCE node
I20260928 08:59:01.840809 4288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:01.841053 4288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:01.842217 4288 hybrid_clock.cc:648] HybridClock initialized: now 1790585941842211 us; error 37 us; skew 500 ppm
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:01.844864 4288 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:59:01.845233 4288 webserver.cc:460] Webserver started at http://127.0.98.3:42263/ using document root <none> and password file <none>
I20260928 08:59:01.845412 4288 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:01.845453 4288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:01.845539 4288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:01.846513 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "f5376206281a4e17b254b105f5ea6c59"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "ea996cea5d08a70734b9e602d9a32b35"
server_key_iv: "fe6fcac2bc9bfe4a7524ba793d8579d5"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.846809 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f5376206281a4e17b254b105f5ea6c59"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "ea996cea5d08a70734b9e602d9a32b35"
server_key_iv: "fe6fcac2bc9bfe4a7524ba793d8579d5"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.848125 4288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:01.848858 4303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.849044 4288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:01.849130 4288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "f5376206281a4e17b254b105f5ea6c59"
format_stamp: "Formatted at 2026-09-28 08:59:01 on dist-test-slave-286n"
server_key: "ea996cea5d08a70734b9e602d9a32b35"
server_key_iv: "fe6fcac2bc9bfe4a7524ba793d8579d5"
server_key_version: "encryptionkey@0"
I20260928 08:59:01.849182 4288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:01.858090 4288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:01.860319 4288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:01.860472 4288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:01.860751 4288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:01.861071 4288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:01.861104 4288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.861125 4288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:01.861152 4288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:01.866770 4288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33553
I20260928 08:59:01.866835 4416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33553 every 8 connection(s)
I20260928 08:59:01.867144 4288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 08:59:01.873121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4288
I20260928 08:59:01.873275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:01.873454 392 external_mini_cluster.cc:1428] Setting key c0b346c077228d2d1e93cc28f389011f
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:01.876768 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.867905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48819 (local address 127.0.98.62:33489)
0928 08:59:01.867960 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:01.867962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:01.868183 (+ 221us) server_negotiation.cc:407] Connection header received
0928 08:59:01.869218 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:01.869219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:01.869231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:01.869277 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:01.869883 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.870399 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.871402 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.871614 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.871699 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.874328 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:01.874329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:01.874330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:01.874337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:01.875527 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.875908 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.875909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.875909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.875964 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.876213 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.876214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.876214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.876326 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:01.876391 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:01.876579 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 08:59:01.876688 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:59:01.876943 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.867881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33489 (local address 127.0.98.3:48819)
0928 08:59:01.868050 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:01.868093 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:01.869105 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:01.869358 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:01.869361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:01.869611 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:01.869790 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:01.869794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.870493 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.870494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:01.871298 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:01.871300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.871708 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.871709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:01.871826 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.872364 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:01.872373 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:01.874175 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:01.875600 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:01.875606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:01.875607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:01.875838 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:01.876030 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:01.876031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:01.876031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:01.876151 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:01.876382 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:01.876384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:01.876487 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 08:59:01.876557 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 08:59:01.876719 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 08:59:01.877385 4417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33489
I20260928 08:59:01.877503 4417 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:01.877684 4417 heartbeater.cc:511] Master 127.0.98.62:33489 requested a full tablet report, sending...
I20260928 08:59:01.878063 3958 ts_manager.cc:194] Registered new tserver with Master: f5376206281a4e17b254b105f5ea6c59 (127.0.98.3:33553)
I20260928 08:59:01.878392 3958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48819
I20260928 08:59:01.885192 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:59:01.907889 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.900664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.0.98.1:41385)
0928 08:59:01.900823 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:01.900828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:01.900847 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:59:01.900874 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:01.900876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:01.900889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:01.900962 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:01.901513 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.902233 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.902931 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.903234 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.903286 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.905481 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:01.905489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:01.905490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:01.905496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:01.906860 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.907180 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.907182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.907183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.907230 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.907387 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.907388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.907388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.907507 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:01.907604 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:01.907674 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 08:59:01.907747 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[3913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:01.915583 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:01.909251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.0.98.62:33489)
0928 08:59:01.909309 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:01.909311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:01.909326 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:59:01.909346 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:01.909347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:01.909355 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:01.909401 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:01.909842 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.910324 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.910944 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:01.911125 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:01.911173 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:01.913256 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:01.913257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:01.913257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:01.913263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:01.914471 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.914800 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.914801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.914802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.914864 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:01.915096 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:01.915097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:01.915097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:01.915258 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:01.915341 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:01.915441 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 08:59:01.915517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 08:59:01.915851 3958 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41848
I20260928 08:59:01.916262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4020
I20260928 08:59:01.921705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4154
I20260928 08:59:01.926127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4288
I20260928 08:59:01.930388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 3926
2026-09-28T08:59:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1849 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[4429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[4429](info): set up 2 sockets
Sep 28 08:59:01 dist-test-slave-286n krb5kdc[4429](info): commencing operation
krb5kdc: starting...
W20260928 08:59:02.964674 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:59:03.083859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:03.168309 4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:03.168524 4442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:03.168550 4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:03.169728 4442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:03.169777 4442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:03.169795 4442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:03.169809 4442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:03.169824 4442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:03.171247 4442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:03.171442 4442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:03.171614 4442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:03.176499 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.176510 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.176510 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:03.176527 4442 server_base.cc:1034] running on GCE node
I20260928 08:59:03.177006 4442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:03.177294 4442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:03.178460 4442 hybrid_clock.cc:648] HybridClock initialized: now 1790585943178426 us; error 56 us; skew 500 ppm
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:03.180943 4442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:59:03.181284 4442 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:03.181355 4442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:03.181511 4442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:03.182636 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "534b8710181540c985dc5279757e5576"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "85233f17656700fee28ae6b62f617212"
server_key_iv: "48ec47004673e1e9afb561285a7d0930"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.182998 4442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "534b8710181540c985dc5279757e5576"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "85233f17656700fee28ae6b62f617212"
server_key_iv: "48ec47004673e1e9afb561285a7d0930"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.184255 4442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 08:59:03.185055 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.185214 4442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.185304 4442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "534b8710181540c985dc5279757e5576"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "85233f17656700fee28ae6b62f617212"
server_key_iv: "48ec47004673e1e9afb561285a7d0930"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.185386 4442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:03.192373 4442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:03.194136 4442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:03.194288 4442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:03.198150 4442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44187
I20260928 08:59:03.198221 4508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44187 every 8 connection(s)
I20260928 08:59:03.198472 4442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:03.198720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4442
I20260928 08:59:03.198828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:03.198987 392 external_mini_cluster.cc:1428] Setting key af09153d4f4d2ad4c8a0cc9c054b5838
I20260928 08:59:03.199630 4509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:03.202093 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576: Bootstrap starting.
I20260928 08:59:03.202714 4509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:03.202991 4509 log.cc:824] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:03.203637 4509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576: No bootstrap required, opened a new log
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:03.206054 4509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b8710181540c985dc5279757e5576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44187 } }
I20260928 08:59:03.206207 4509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:03.206257 4509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534b8710181540c985dc5279757e5576, State: Initialized, Role: FOLLOWER
I20260928 08:59:03.206427 4509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b8710181540c985dc5279757e5576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44187 } }
I20260928 08:59:03.206521 4509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:03.206563 4509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:03.206614 4509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:03.207103 4509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b8710181540c985dc5279757e5576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44187 } }
I20260928 08:59:03.207219 4509 leader_election.cc:302] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 534b8710181540c985dc5279757e5576; no voters:
I20260928 08:59:03.207376 4509 leader_election.cc:288] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:03.207402 4514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:03.207509 4514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [term 1 LEADER]: Becoming Leader. State: Replica: 534b8710181540c985dc5279757e5576, State: Running, Role: LEADER
I20260928 08:59:03.207587 4514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b8710181540c985dc5279757e5576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44187 } }
I20260928 08:59:03.207629 4509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:03.207914 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "534b8710181540c985dc5279757e5576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b8710181540c985dc5279757e5576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44187 } } }
I20260928 08:59:03.208067 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:03.208268 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 534b8710181540c985dc5279757e5576. Latest consensus state: current_term: 1 leader_uuid: "534b8710181540c985dc5279757e5576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "534b8710181540c985dc5279757e5576" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44187 } } }
I20260928 08:59:03.208384 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:03.208734 4522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:03.209268 4522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:03.209860 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.199477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.98.62:44187)
0928 08:59:03.199671 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.199682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.199721 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:59:03.200167 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.200174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.200218 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.200361 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:03.201047 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.201920 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.202763 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.203246 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.203373 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.205919 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:03.205924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:03.205924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:03.205942 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:03.208067 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.208468 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.208470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.208470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.208544 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.208819 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.208819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.208820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.209212 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:03.209378 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.209486 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.209707 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260928 08:59:03.210809 4522 catalog_manager.cc:1347] Generated new cluster ID: 3e23aadc1809434c8d1214551582b9a8
I20260928 08:59:03.210848 4522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:03.217219 4522 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:03.217736 4522 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:03.219671 4522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576: Generated new TSK 0
I20260928 08:59:03.219880 4522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:59:03.253139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44187
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:03.333952 4535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:03.334161 4535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:03.334193 4535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:03.335304 4535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:03.335340 4535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:03.335366 4535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:03.336920 4535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:03.337109 4535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:03.337378 4535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:03.342420 4543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.342470 4541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.342437 4540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:03.342737 4535 server_base.cc:1034] running on GCE node
I20260928 08:59:03.342980 4535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:03.343243 4535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:03.344434 4535 hybrid_clock.cc:648] HybridClock initialized: now 1790585943344419 us; error 30 us; skew 500 ppm
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:03.347225 4535 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:59:03.347631 4535 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:03.347692 4535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:03.347877 4535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:03.349092 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "54b04ea2f444415b84a3ee668f0488c3"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "d865d129d4cab4be1a7e44df8b858c4f"
server_key_iv: "b54f3d0915fce76bf7ad3e6c149d7290"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.349480 4535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54b04ea2f444415b84a3ee668f0488c3"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "d865d129d4cab4be1a7e44df8b858c4f"
server_key_iv: "b54f3d0915fce76bf7ad3e6c149d7290"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.350925 4535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:59:03.351786 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.352003 4535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:03.352098 4535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "54b04ea2f444415b84a3ee668f0488c3"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "d865d129d4cab4be1a7e44df8b858c4f"
server_key_iv: "b54f3d0915fce76bf7ad3e6c149d7290"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.352164 4535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:03.369992 4535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:03.372411 4535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:03.372529 4535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:03.372786 4535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:03.373126 4535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:03.373183 4535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.373229 4535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:03.373260 4535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.378919 4535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43505
I20260928 08:59:03.378991 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43505 every 8 connection(s)
I20260928 08:59:03.379269 4535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:03.388031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4535
I20260928 08:59:03.388139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:03.388363 392 external_mini_cluster.cc:1428] Setting key f24ffb03fee09e9430546ef5a1afa665
I20260928 08:59:03.388558 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.379983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56087 (local address 127.0.98.62:44187)
0928 08:59:03.380047 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.380049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.380255 (+ 206us) server_negotiation.cc:407] Connection header received
0928 08:59:03.380830 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.380831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.380841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.380891 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:03.381779 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.382271 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.383244 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.383434 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.383483 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.385628 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:03.385630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:03.385630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:03.385638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:03.387032 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.387455 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.387456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.387457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.387514 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.387767 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.387768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.387768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.387928 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:03.388003 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.388370 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.388462 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:59:03.388794 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.379975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44187 (local address 127.0.98.1:56087)
0928 08:59:03.380130 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.380164 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.380727 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.380952 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.380955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.381209 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:03.381696 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.381708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.382353 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.382354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.383167 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.383169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.383503 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.383504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.383600 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.384096 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:03.384103 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:03.385530 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:03.387130 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:03.387137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:03.387140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:03.387362 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:03.387579 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:03.387580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:03.387580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:03.387697 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:03.388005 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:03.388011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:03.388169 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.388329 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.388500 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260928 08:59:03.389282 4663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44187
I20260928 08:59:03.389394 4663 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:03.389557 4663 heartbeater.cc:511] Master 127.0.98.62:44187 requested a full tablet report, sending...
I20260928 08:59:03.390026 4473 ts_manager.cc:194] Registered new tserver with Master: 54b04ea2f444415b84a3ee668f0488c3 (127.0.98.1:43505)
I20260928 08:59:03.390614 4473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56087
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:59:03.415067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44187
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:03.494854 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:03.495082 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:03.495123 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:03.496285 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:03.496348 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:03.496394 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:03.497896 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:03.498150 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:03.498374 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:03.503531 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.503597 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.503538 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:03.503875 4668 server_base.cc:1034] running on GCE node
I20260928 08:59:03.504096 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:03.504340 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:03.505506 4668 hybrid_clock.cc:648] HybridClock initialized: now 1790585943505498 us; error 35 us; skew 500 ppm
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:03.508159 4668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:59:03.508678 4668 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:03.508749 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:03.508888 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:03.510043 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "6d5027a137fe45339b9a117a7d383759"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "b967e943cd2f04eb0a6a44b6b0bfb0cf"
server_key_iv: "47ba5d43fedef7c11a0cd927c6faf288"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.510403 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6d5027a137fe45339b9a117a7d383759"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "b967e943cd2f04eb0a6a44b6b0bfb0cf"
server_key_iv: "47ba5d43fedef7c11a0cd927c6faf288"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.511708 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:03.512420 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.512583 4668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:03.512698 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "6d5027a137fe45339b9a117a7d383759"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "b967e943cd2f04eb0a6a44b6b0bfb0cf"
server_key_iv: "47ba5d43fedef7c11a0cd927c6faf288"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.512797 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:03.530333 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:03.532549 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:03.532780 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:03.533048 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:03.533388 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:03.533444 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.533488 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:03.533519 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.539680 4668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45693
I20260928 08:59:03.539754 4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45693 every 8 connection(s)
I20260928 08:59:03.540040 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:03.549178 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.540817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56647 (local address 127.0.98.62:44187)
0928 08:59:03.540900 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.540902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.541136 (+ 234us) server_negotiation.cc:407] Connection header received
0928 08:59:03.541706 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.541707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.541716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.541786 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:03.542707 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.543195 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.544248 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.544427 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.544481 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.546720 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:03.546722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:03.546722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:03.546730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:03.547967 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.548339 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.548339 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.548340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.548389 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.548591 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.548592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.548592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.548749 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:03.548803 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.549048 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.549111 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:59:03.549440 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.540826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44187 (local address 127.0.98.2:56647)
0928 08:59:03.540994 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.541039 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.541610 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.541846 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.541849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.542115 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:03.542621 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.542627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.543349 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.543350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.544167 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.544169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.544522 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.544524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.544697 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.545216 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:03.545231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:03.546630 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:03.548044 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:03.548051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:03.548052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:03.548268 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:03.548445 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:03.548446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:03.548446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:03.548541 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:03.548810 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:03.548814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:03.548935 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.549022 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.549144 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260928 08:59:03.549752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4668
I20260928 08:59:03.549865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:03.549986 4796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44187
I20260928 08:59:03.550087 4796 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:03.550098 392 external_mini_cluster.cc:1428] Setting key 934dc369e7052ec120406e9c9a959ae5
I20260928 08:59:03.550244 4796 heartbeater.cc:511] Master 127.0.98.62:44187 requested a full tablet report, sending...
I20260928 08:59:03.550698 4472 ts_manager.cc:194] Registered new tserver with Master: 6d5027a137fe45339b9a117a7d383759 (127.0.98.2:45693)
I20260928 08:59:03.551072 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56647
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:59:03.575399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44187
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:03.656148 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:03.656450 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:03.656494 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:03.657694 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:03.657763 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:03.657810 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:03.659380 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:03.659662 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:03.659909 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:03.665057 4809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.665037 4807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:03.665036 4806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:03.665256 4801 server_base.cc:1034] running on GCE node
I20260928 08:59:03.665524 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:03.665774 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:03.666934 4801 hybrid_clock.cc:648] HybridClock initialized: now 1790585943666933 us; error 42 us; skew 500 ppm
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:03.669564 4801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:59:03.669943 4801 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:03.670017 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:03.670172 4801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:03.671226 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "d84df96507594a07ac10fccb35ae0ac9"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "01ab70e6a530ca08c07f9faa8a00ccaa"
server_key_iv: "1486bb9f196f8848cab1e741e2151f51"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.671595 4801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d84df96507594a07ac10fccb35ae0ac9"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "01ab70e6a530ca08c07f9faa8a00ccaa"
server_key_iv: "1486bb9f196f8848cab1e741e2151f51"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.673012 4801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:03.673780 4815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.673949 4801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:03.674052 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "d84df96507594a07ac10fccb35ae0ac9"
format_stamp: "Formatted at 2026-09-28 08:59:03 on dist-test-slave-286n"
server_key: "01ab70e6a530ca08c07f9faa8a00ccaa"
server_key_iv: "1486bb9f196f8848cab1e741e2151f51"
server_key_version: "encryptionkey@0"
I20260928 08:59:03.674124 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:03.684563 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:03.686810 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:03.686973 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:03.687202 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:03.687536 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:03.687594 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.687633 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:03.687665 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:03.693631 4801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42511
I20260928 08:59:03.693710 4928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42511 every 8 connection(s)
I20260928 08:59:03.693985 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:59:03 dist-test-slave-286n krb5kdc[4429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:03.700373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4801
I20260928 08:59:03.700491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:03.700790 392 external_mini_cluster.cc:1428] Setting key 2b815acc8f1ae022ea55b580a02ae680
I20260928 08:59:03.703583 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.694740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58919 (local address 127.0.98.62:44187)
0928 08:59:03.694807 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.694809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.695001 (+ 192us) server_negotiation.cc:407] Connection header received
0928 08:59:03.695561 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.695562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.695572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.695655 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:03.696583 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.697146 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.698145 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.698345 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.698403 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.700578 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:03.700580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:03.700580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:03.700588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:03.702171 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.702587 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.702588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.702589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.702654 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.702919 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.702919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.702920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.703040 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:03.703113 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.703391 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.703493 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:59:03.703774 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.694720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44187 (local address 127.0.98.3:58919)
0928 08:59:03.694893 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.694930 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.695448 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.695711 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.695714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.695988 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:03.696488 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.696493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.697222 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.697223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.698055 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.698057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.698419 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.698420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.698519 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.699026 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:03.699035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:03.700441 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:03.702271 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:03.702278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:03.702280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:03.702510 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:03.702719 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:03.702720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:03.702720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:03.702839 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:03.703103 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:03.703109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:03.703255 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.703372 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.703537 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260928 08:59:03.704236 4929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44187
I20260928 08:59:03.704346 4929 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:03.704468 4929 heartbeater.cc:511] Master 127.0.98.62:44187 requested a full tablet report, sending...
I20260928 08:59:03.704846 4472 ts_manager.cc:194] Registered new tserver with Master: d84df96507594a07ac10fccb35ae0ac9 (127.0.98.3:42511)
I20260928 08:59:03.705179 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58919
I20260928 08:59:03.712672 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:59:03.720358 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.713964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.98.62:44187)
0928 08:59:03.714045 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.714047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.714129 (+ 82us) server_negotiation.cc:407] Connection header received
0928 08:59:03.714214 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.714214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.714224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.714273 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:03.714858 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.715364 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.716142 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.716356 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.716449 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.717785 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:03.717786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:03.717787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:03.717794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:03.718997 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.719387 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.719388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.719389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.719445 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:03.719654 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:03.719655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:03.719655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:03.719891 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:03.720008 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.720165 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.720255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:59:03.722168 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:59:03.723773 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:03.730310 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.726286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.98.1:43505)
0928 08:59:03.726501 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.726505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.726561 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:59:03.726648 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.726651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.726667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.726749 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:03.726943 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.727808 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.729259 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.729901 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.729994 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.730056 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.730181 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.730219 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":136,"threads_started":1}
I20260928 08:59:03.730301 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.726126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43505 (local address 127.0.0.1:60736)
0928 08:59:03.726507 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.726517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.726574 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.726754 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.726757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.726759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:03.726843 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.726845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.727894 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.727894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.729169 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.729172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.729998 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.729999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.730079 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.730092 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.730163 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.730220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":80,"threads_started":1}
I20260928 08:59:03.730501 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.726205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.0.98.2:45693)
0928 08:59:03.726694 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.726698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.726707 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:03.726751 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.726752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.726768 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.726842 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:03.726958 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.728038 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.729312 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.729955 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.730025 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.730077 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.730346 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.730394 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":144,"threads_started":1}
I20260928 08:59:03.730659 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.726101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45693 (local address 127.0.0.1:40544)
0928 08:59:03.726238 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.726282 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.726349 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.726849 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.726850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.726852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:03.726892 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.726893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.728136 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.728137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.729243 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.729245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.730114 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.730115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.730179 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.730183 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.730233 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.730275 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260928 08:59:03.731128 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.726298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42511 (local address 127.0.0.1:33100)
0928 08:59:03.726518 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.726528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.726578 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.727266 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.727267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.727269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:03.727332 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.727333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.728448 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.728449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.729507 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.729510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.730883 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.730884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.730955 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.730959 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.731026 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.731066 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260928 08:59:03.731166 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.726502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.98.3:42511)
0928 08:59:03.727090 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.727093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.727128 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:59:03.727164 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.727165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.727179 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.727265 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:03.727393 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.728354 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.729595 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.730300 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.730783 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.730856 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.731043 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.731085 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":80,"threads_started":1}
I20260928 08:59:03.731122 4597 tablet_service.cc:1467] Processing CreateTablet for tablet c9db4a24eeb742c595c8e760b8ea6067 (DEFAULT_TABLE table=test-table [id=4ec8b384be8442ae997022317bb5703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:03.731351 4730 tablet_service.cc:1467] Processing CreateTablet for tablet c9db4a24eeb742c595c8e760b8ea6067 (DEFAULT_TABLE table=test-table [id=4ec8b384be8442ae997022317bb5703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:03.731473 4597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9db4a24eeb742c595c8e760b8ea6067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:03.731626 4730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9db4a24eeb742c595c8e760b8ea6067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:03.731689 4863 tablet_service.cc:1467] Processing CreateTablet for tablet c9db4a24eeb742c595c8e760b8ea6067 (DEFAULT_TABLE table=test-table [id=4ec8b384be8442ae997022317bb5703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:03.731914 4863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9db4a24eeb742c595c8e760b8ea6067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:03.734413 4946 tablet_bootstrap.cc:492] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Bootstrap starting.
I20260928 08:59:03.734724 4947 tablet_bootstrap.cc:492] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Bootstrap starting.
I20260928 08:59:03.735306 4946 tablet_bootstrap.cc:654] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:03.735317 4947 tablet_bootstrap.cc:654] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:03.735986 4947 log.cc:824] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:03.735986 4946 log.cc:824] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:03.736337 4948 tablet_bootstrap.cc:492] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: Bootstrap starting.
I20260928 08:59:03.736861 4947 tablet_bootstrap.cc:492] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: No bootstrap required, opened a new log
I20260928 08:59:03.736922 4947 ts_tablet_manager.cc:1397] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:59:03.737008 4948 tablet_bootstrap.cc:654] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:03.737125 4946 tablet_bootstrap.cc:492] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: No bootstrap required, opened a new log
I20260928 08:59:03.737201 4946 ts_tablet_manager.cc:1397] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:59:03.737612 4948 log.cc:824] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:03.738344 4948 tablet_bootstrap.cc:492] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: No bootstrap required, opened a new log
I20260928 08:59:03.738412 4948 ts_tablet_manager.cc:1397] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:59:03.738852 4947 raft_consensus.cc:348] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.739056 4947 raft_consensus.cc:374] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:03.739101 4947 raft_consensus.cc:729] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5027a137fe45339b9a117a7d383759, State: Initialized, Role: FOLLOWER
I20260928 08:59:03.739217 4947 consensus_queue.cc:260] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.739395 4946 raft_consensus.cc:348] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.739527 4947 ts_tablet_manager.cc:1428] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:59:03.739547 4946 raft_consensus.cc:374] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:03.739691 4946 raft_consensus.cc:729] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84df96507594a07ac10fccb35ae0ac9, State: Initialized, Role: FOLLOWER
I20260928 08:59:03.739812 4946 consensus_queue.cc:260] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.740084 4946 ts_tablet_manager.cc:1428] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:59:03.740104 4929 heartbeater.cc:503] Master 127.0.98.62:44187 was elected leader, sending a full tablet report...
I20260928 08:59:03.740476 4948 raft_consensus.cc:348] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.740599 4948 raft_consensus.cc:374] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:03.740620 4948 raft_consensus.cc:729] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b04ea2f444415b84a3ee668f0488c3, State: Initialized, Role: FOLLOWER
I20260928 08:59:03.740736 4796 heartbeater.cc:503] Master 127.0.98.62:44187 was elected leader, sending a full tablet report...
I20260928 08:59:03.740765 4948 consensus_queue.cc:260] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.741031 4948 ts_tablet_manager.cc:1428] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 08:59:03.741044 4663 heartbeater.cc:503] Master 127.0.98.62:44187 was elected leader, sending a full tablet report...
W20260928 08:59:03.790601 4797 tablet.cc:2369] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:59:03.861856 4952 raft_consensus.cc:482] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:03.861975 4952 raft_consensus.cc:504] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.862413 4952 leader_election.cc:288] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d84df96507594a07ac10fccb35ae0ac9 (127.0.98.3:42511), 54b04ea2f444415b84a3ee668f0488c3 (127.0.98.1:43505)
I20260928 08:59:03.865227 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.862656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54429 (local address 127.0.98.1:43505)
0928 08:59:03.862705 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.862707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.862719 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:59:03.862806 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.862807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.862819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.862896 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:03.863054 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.863554 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.864494 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.864876 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.864965 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.864975 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.865132 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.865167 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:59:03.865227 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.862589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43505 (local address 127.0.98.2:54429)
0928 08:59:03.862662 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.862675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.862718 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.862900 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.862900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.862902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:03.862981 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.862983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.863646 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.863647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.864400 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.864402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.864965 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.864966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.865050 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.865053 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.865114 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.865153 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260928 08:59:03.865371 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.862668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48175 (local address 127.0.98.3:42511)
0928 08:59:03.862730 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.862732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.862867 (+ 135us) server_negotiation.cc:407] Connection header received
0928 08:59:03.862936 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.862937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.862948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.863032 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:03.863175 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.863631 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.864553 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.864990 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.865081 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.865090 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.865243 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.865281 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:59:03.865546 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.862608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42511 (local address 127.0.98.2:48175)
0928 08:59:03.862795 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.862804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.862881 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.863034 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:03.863034 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:03.863036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:03.863108 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.863109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.863717 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.863718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.864455 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:03.864456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.865085 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.865086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:03.865161 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.865164 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:03.865236 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 08:59:03.865415 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":97,"threads_started":1}
I20260928 08:59:03.865540 4617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9db4a24eeb742c595c8e760b8ea6067" candidate_uuid: "6d5027a137fe45339b9a117a7d383759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b04ea2f444415b84a3ee668f0488c3" is_pre_election: true
I20260928 08:59:03.865696 4617 raft_consensus.cc:2454] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5027a137fe45339b9a117a7d383759 in term 0.
I20260928 08:59:03.865913 4683 leader_election.cc:302] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54b04ea2f444415b84a3ee668f0488c3, 6d5027a137fe45339b9a117a7d383759; no voters:
I20260928 08:59:03.865904 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9db4a24eeb742c595c8e760b8ea6067" candidate_uuid: "6d5027a137fe45339b9a117a7d383759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84df96507594a07ac10fccb35ae0ac9" is_pre_election: true
I20260928 08:59:03.866022 4952 raft_consensus.cc:2781] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:03.866043 4883 raft_consensus.cc:2454] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5027a137fe45339b9a117a7d383759 in term 0.
I20260928 08:59:03.866074 4952 raft_consensus.cc:482] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:03.866104 4952 raft_consensus.cc:3037] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:03.866770 4952 raft_consensus.cc:504] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.866916 4952 leader_election.cc:288] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [CANDIDATE]: Term 1 election: Requested vote from peers d84df96507594a07ac10fccb35ae0ac9 (127.0.98.3:42511), 54b04ea2f444415b84a3ee668f0488c3 (127.0.98.1:43505)
I20260928 08:59:03.867031 4883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9db4a24eeb742c595c8e760b8ea6067" candidate_uuid: "6d5027a137fe45339b9a117a7d383759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84df96507594a07ac10fccb35ae0ac9"
I20260928 08:59:03.867106 4883 raft_consensus.cc:3037] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:03.867110 4617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9db4a24eeb742c595c8e760b8ea6067" candidate_uuid: "6d5027a137fe45339b9a117a7d383759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b04ea2f444415b84a3ee668f0488c3"
I20260928 08:59:03.867187 4617 raft_consensus.cc:3037] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:03.867781 4883 raft_consensus.cc:2454] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5027a137fe45339b9a117a7d383759 in term 1.
I20260928 08:59:03.867993 4617 raft_consensus.cc:2454] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5027a137fe45339b9a117a7d383759 in term 1.
I20260928 08:59:03.868001 4684 leader_election.cc:302] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5027a137fe45339b9a117a7d383759, d84df96507594a07ac10fccb35ae0ac9; no voters:
I20260928 08:59:03.868088 4952 raft_consensus.cc:2781] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:03.868283 4952 raft_consensus.cc:686] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 1 LEADER]: Becoming Leader. State: Replica: 6d5027a137fe45339b9a117a7d383759, State: Running, Role: LEADER
I20260928 08:59:03.868383 4952 consensus_queue.cc:237] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } }
I20260928 08:59:03.869194 4470 catalog_manager.cc:5495] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d5027a137fe45339b9a117a7d383759 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6d5027a137fe45339b9a117a7d383759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d5027a137fe45339b9a117a7d383759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:03.878657 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.876480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.98.2:45693)
0928 08:59:03.876518 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:03.876520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:03.876557 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:59:03.876582 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:03.876583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:03.876598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:03.876729 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:03.876857 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.877285 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.878001 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:03.878202 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:03.878253 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:03.878417 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:03.878506 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:03.878564 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:03.878576 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 08:59:03.878592 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 08:59:03.879971 4664 tablet.cc:2369] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:59:03.880312 4617 raft_consensus.cc:1261] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 1 FOLLOWER]: Refusing update from remote peer 6d5027a137fe45339b9a117a7d383759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:03.880331 4883 raft_consensus.cc:1261] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 1 FOLLOWER]: Refusing update from remote peer 6d5027a137fe45339b9a117a7d383759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:03.880662 4952 consensus_queue.cc:1035] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84df96507594a07ac10fccb35ae0ac9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:03.880802 4956 consensus_queue.cc:1035] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b04ea2f444415b84a3ee668f0488c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:03.883708 4961 mvcc.cc:205] Tried to move back new op lower bound from 7334240026130980864 to 7334240026085236736. Current Snapshot: MvccSnapshot[applied={T|T < 7334240026130980864}]
I20260928 08:59:03.885429 4470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:03.885560 4470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:03.886252 4470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 534b8710181540c985dc5279757e5576: Sending DeleteTablet for 3 replicas of tablet c9db4a24eeb742c595c8e760b8ea6067
I20260928 08:59:03.886572 4730 tablet_service.cc:1514] Processing DeleteTablet for tablet c9db4a24eeb742c595c8e760b8ea6067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40544
I20260928 08:59:03.886613 4597 tablet_service.cc:1514] Processing DeleteTablet for tablet c9db4a24eeb742c595c8e760b8ea6067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60736
I20260928 08:59:03.886667 4863 tablet_service.cc:1514] Processing DeleteTablet for tablet c9db4a24eeb742c595c8e760b8ea6067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33100
I20260928 08:59:03.886826 4968 tablet_replica.cc:330] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: stopping tablet replica
I20260928 08:59:03.886924 4968 raft_consensus.cc:2229] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:59:03.886952 4969 tablet_replica.cc:330] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3: stopping tablet replica
I20260928 08:59:03.887022 4969 raft_consensus.cc:2229] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:03.887045 4968 raft_consensus.cc:2258] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:03.887102 4969 raft_consensus.cc:2258] T c9db4a24eeb742c595c8e760b8ea6067 P 54b04ea2f444415b84a3ee668f0488c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:03.887144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4535
W20260928 08:59:03.892477 4457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43505 (error 108)
I20260928 08:59:03.892630 4970 tablet_replica.cc:330] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: stopping tablet replica
I20260928 08:59:03.892753 4968 ts_tablet_manager.cc:1905] T c9db4a24eeb742c595c8e760b8ea6067 P 6d5027a137fe45339b9a117a7d383759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:03.892916 4970 raft_consensus.cc:2229] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:03.893081 4970 raft_consensus.cc:2258] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 08:59:03.893160 4941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:03.892941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43505 (local address 127.0.0.1:60742)
0928 08:59:03.892989 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.893003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:03.893065 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:03.893111 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43505: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 08:59:03.893276 4457 catalog_manager.cc:4584] TS 54b04ea2f444415b84a3ee668f0488c3 (127.0.98.1:43505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9db4a24eeb742c595c8e760b8ea6067: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43505: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 08:59:03.893338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4668
I20260928 08:59:03.895200 4970 ts_tablet_manager.cc:1905] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:03.896770 4970 ts_tablet_manager.cc:1918] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:03.896833 4970 log.cc:1196] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/c9db4a24eeb742c595c8e760b8ea6067
I20260928 08:59:03.897100 4970 ts_tablet_manager.cc:1939] T c9db4a24eeb742c595c8e760b8ea6067 P d84df96507594a07ac10fccb35ae0ac9: Deleting consensus metadata
I20260928 08:59:03.897536 4458 catalog_manager.cc:4841] TS d84df96507594a07ac10fccb35ae0ac9 (127.0.98.3:42511): tablet c9db4a24eeb742c595c8e760b8ea6067 (table test-table [id=4ec8b384be8442ae997022317bb5703c]) successfully deleted
I20260928 08:59:03.898373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4801
I20260928 08:59:03.902691 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:03.898484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45693 (local address 127.0.0.1:40552)
0928 08:59:03.902624 (+ 4140us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:03.902662 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4118}
I20260928 08:59:03.902871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4442
2026-09-28T08:59:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1971 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T08:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:03Z Disabled control of system clock
I20260928 08:59:03.929723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39955
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:59:04.010712 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:04.010913 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:04.010933 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:04.011986 4975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:04.012017 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:04.012029 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:04.012041 4975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:04.012053 4975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:04.013330 4975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:04.013510 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:04.013731 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:04.018497 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.018520 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.018519 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.018844 4975 server_base.cc:1034] running on GCE node
I20260928 08:59:04.019052 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:04.019280 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:04.020421 4975 hybrid_clock.cc:648] HybridClock initialized: now 1790585944020406 us; error 27 us; skew 500 ppm
I20260928 08:59:04.021639 4975 webserver.cc:460] Webserver started at http://127.0.98.62:39267/ using document root <none> and password file <none>
I20260928 08:59:04.021889 4975 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:04.021934 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:04.022082 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:04.023152 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "2ca8b5d0170744f3af279f26c7d0d890"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "99efd6764731f37f67c062e17cd942c3"
server_key_iv: "2d076b01837c076fe46d2fce8d1436ef"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.023500 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ca8b5d0170744f3af279f26c7d0d890"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "99efd6764731f37f67c062e17cd942c3"
server_key_iv: "2d076b01837c076fe46d2fce8d1436ef"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.024816 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:04.025523 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.025696 4975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:04.025795 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "2ca8b5d0170744f3af279f26c7d0d890"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "99efd6764731f37f67c062e17cd942c3"
server_key_iv: "2d076b01837c076fe46d2fce8d1436ef"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.025875 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:04.029219 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:04.029503 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:04.029646 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:04.033573 4975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39955
I20260928 08:59:04.033638 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39955 every 8 connection(s)
I20260928 08:59:04.033978 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:04.034600 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:04.037374 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890: Bootstrap starting.
I20260928 08:59:04.037909 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:04.038141 5042 log.cc:824] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:04.038830 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890: No bootstrap required, opened a new log
I20260928 08:59:04.040467 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca8b5d0170744f3af279f26c7d0d890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260928 08:59:04.040616 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:04.040688 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca8b5d0170744f3af279f26c7d0d890, State: Initialized, Role: FOLLOWER
I20260928 08:59:04.040834 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca8b5d0170744f3af279f26c7d0d890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260928 08:59:04.040910 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:04.040940 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:04.040987 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:04.041488 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca8b5d0170744f3af279f26c7d0d890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260928 08:59:04.041599 5042 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca8b5d0170744f3af279f26c7d0d890; no voters:
I20260928 08:59:04.041760 5042 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:04.041831 5045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:04.042004 5045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca8b5d0170744f3af279f26c7d0d890, State: Running, Role: LEADER
I20260928 08:59:04.042024 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:04.042091 5045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca8b5d0170744f3af279f26c7d0d890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260928 08:59:04.042747 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca8b5d0170744f3af279f26c7d0d890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca8b5d0170744f3af279f26c7d0d890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } }
I20260928 08:59:04.042855 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:04.043005 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca8b5d0170744f3af279f26c7d0d890. Latest consensus state: current_term: 1 leader_uuid: "2ca8b5d0170744f3af279f26c7d0d890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca8b5d0170744f3af279f26c7d0d890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } }
I20260928 08:59:04.043047 5055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:04.043068 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:04.043627 5055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:04.043627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 4975
I20260928 08:59:04.043718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:04.043910 392 external_mini_cluster.cc:1428] Setting key b3c5fc5c6d1bd9554dea48cb56f368e9
I20260928 08:59:04.045274 5055 catalog_manager.cc:1347] Generated new cluster ID: 13ff988cc75744039b5169fad82334c9
I20260928 08:59:04.045326 5055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:04.049520 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.044378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.0.98.62:39955)
0928 08:59:04.044784 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.044792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.044825 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:59:04.044956 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.044965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.045009 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.045159 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.049061 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.049064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.049065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.049068 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.049101 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.049113 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.049180 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.049293 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.049348 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":77,"threads_started":1}
I20260928 08:59:04.064576 5055 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:04.064806 5055 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:04.068176 5055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890: Generated new TSK 0
I20260928 08:59:04.068375 5055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:59:04.073786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--builtin_ntp_servers=127.0.98.20:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:59:04.168622 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:04.168895 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:04.168919 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:04.170130 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:04.170169 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:04.170197 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:04.171543 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:04.171780 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:04.172082 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:04.177946 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.178092 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.178128 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.178251 5066 server_base.cc:1034] running on GCE node
I20260928 08:59:04.178436 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:04.178686 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:04.179872 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790585944179845 us; error 40 us; skew 500 ppm
I20260928 08:59:04.181306 5066 webserver.cc:460] Webserver started at http://127.0.98.1:43411/ using document root <none> and password file <none>
I20260928 08:59:04.181551 5066 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:04.181622 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:04.181788 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:04.182847 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "30926bdda611ce7828e852d7c5342c58"
server_key_iv: "b698b7ab6f6bbcd91cf4ad75e4c216b5"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.183211 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "30926bdda611ce7828e852d7c5342c58"
server_key_iv: "b698b7ab6f6bbcd91cf4ad75e4c216b5"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.184568 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 08:59:04.185415 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.185582 5066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:04.185680 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "30926bdda611ce7828e852d7c5342c58"
server_key_iv: "b698b7ab6f6bbcd91cf4ad75e4c216b5"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.185771 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:04.189077 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:04.189340 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:04.189478 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:04.189685 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:04.189973 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:04.190025 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.190064 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:04.190091 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.195866 5066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43713
I20260928 08:59:04.195940 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43713 every 8 connection(s)
I20260928 08:59:04.196300 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:04.199186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5066
I20260928 08:59:04.199295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:04.199468 392 external_mini_cluster.cc:1428] Setting key 1ab841f78c3be45202c278fdef1e0672
I20260928 08:59:04.201120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--builtin_ntp_servers=127.0.98.20:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:59:04.201550 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.197070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47761 (local address 127.0.98.62:39955)
0928 08:59:04.197118 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.197121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.198828 (+ 1707us) server_negotiation.cc:407] Connection header received
0928 08:59:04.199560 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.199561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.199569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.199684 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.201208 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.201209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.201210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.201216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.201236 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.201242 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.201347 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.201462 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.201480 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:04.201902 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.197061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.98.1:47761)
0928 08:59:04.198668 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.198712 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.199434 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.199769 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.199774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.199785 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.201014 (+ 1229us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.201015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.201052 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.201088 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.201346 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.201352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.201355 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.201633 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.201698 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"thread_start_us":77,"threads_started":1}
I20260928 08:59:04.202327 5194 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39955
I20260928 08:59:04.202418 5194 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:04.202591 5194 heartbeater.cc:511] Master 127.0.98.62:39955 requested a full tablet report, sending...
I20260928 08:59:04.203060 5006 ts_manager.cc:194] Registered new tserver with Master: b8ddb1e0f8ff4725b3f0dfd7e4583f85 (127.0.98.1:43713)
W20260928 08:59:04.283776 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:04.283965 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:04.283987 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:04.285188 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:04.285228 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:04.285285 5197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:04.286661 5197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:04.286896 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:04.287166 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:04.292124 5203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.292181 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.292124 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.292435 5197 server_base.cc:1034] running on GCE node
I20260928 08:59:04.292687 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:04.292936 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:04.294098 5197 hybrid_clock.cc:648] HybridClock initialized: now 1790585944294093 us; error 35 us; skew 500 ppm
I20260928 08:59:04.295392 5197 webserver.cc:460] Webserver started at http://127.0.98.2:39515/ using document root <none> and password file <none>
I20260928 08:59:04.295634 5197 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:04.295701 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:04.295827 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:04.296948 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "434bd76e42134ef9ad1014f9f1c0e20d"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "49f79e1b8b982f2a19167b9db4977cee"
server_key_iv: "5f620d371e1ee59591a5ecfe49d9f8ba"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.297305 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "434bd76e42134ef9ad1014f9f1c0e20d"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "49f79e1b8b982f2a19167b9db4977cee"
server_key_iv: "5f620d371e1ee59591a5ecfe49d9f8ba"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.298549 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:59:04.299263 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.299435 5197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:04.299530 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "434bd76e42134ef9ad1014f9f1c0e20d"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "49f79e1b8b982f2a19167b9db4977cee"
server_key_iv: "5f620d371e1ee59591a5ecfe49d9f8ba"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.299609 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:04.302986 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:04.303248 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:04.303393 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:04.303597 5197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:04.303893 5197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:04.303947 5197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.303987 5197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:04.304013 5197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.309763 5197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42011
I20260928 08:59:04.309798 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42011 every 8 connection(s)
I20260928 08:59:04.310135 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:59:04.313153 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.310898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43307 (local address 127.0.98.62:39955)
0928 08:59:04.310966 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.310968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.311195 (+ 227us) server_negotiation.cc:407] Connection header received
0928 08:59:04.311604 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.311605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.311610 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.311686 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.312944 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.312946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.312946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.312950 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.312966 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.312970 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.313027 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.313088 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.313100 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:59:04.313336 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.310823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.98.2:43307)
0928 08:59:04.311040 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.311079 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.311520 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.311737 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.311740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.311748 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.312777 (+ 1029us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.312778 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.312815 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.312853 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.313034 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.313038 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.313041 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.313096 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.313142 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260928 08:59:04.313818 5325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39955
I20260928 08:59:04.313906 5325 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:04.314013 5325 heartbeater.cc:511] Master 127.0.98.62:39955 requested a full tablet report, sending...
I20260928 08:59:04.314381 5006 ts_manager.cc:194] Registered new tserver with Master: 434bd76e42134ef9ad1014f9f1c0e20d (127.0.98.2:42011)
I20260928 08:59:04.316738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5197
I20260928 08:59:04.316850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:04.317044 392 external_mini_cluster.cc:1428] Setting key 63ddb431a1b20500333c51b79ebd56c4
I20260928 08:59:04.317932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--builtin_ntp_servers=127.0.98.20:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:59:04.398509 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:04.398747 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:04.398782 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:04.399971 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:04.400053 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:04.400099 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:04.401510 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:04.401767 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:04.402119 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:04.407120 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.407152 5336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.407153 5333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.407263 5328 server_base.cc:1034] running on GCE node
I20260928 08:59:04.407591 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:04.407825 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:04.408958 5328 hybrid_clock.cc:648] HybridClock initialized: now 1790585944408950 us; error 28 us; skew 500 ppm
I20260928 08:59:04.410194 5328 webserver.cc:460] Webserver started at http://127.0.98.3:44825/ using document root <none> and password file <none>
I20260928 08:59:04.410429 5328 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:04.410478 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:04.410624 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:04.411664 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "dffe0b77baf84806ae2ccdb4998d401e"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "3dcbc9862387ad095d216037943ecee8"
server_key_iv: "781baa60a7ac2f17dcff6269b9118b4e"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.412014 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dffe0b77baf84806ae2ccdb4998d401e"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "3dcbc9862387ad095d216037943ecee8"
server_key_iv: "781baa60a7ac2f17dcff6269b9118b4e"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.413476 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:04.414299 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.414484 5328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:04.414579 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "dffe0b77baf84806ae2ccdb4998d401e"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "3dcbc9862387ad095d216037943ecee8"
server_key_iv: "781baa60a7ac2f17dcff6269b9118b4e"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.414659 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:04.418035 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:04.418303 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:04.418439 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:04.418635 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:04.418927 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:04.418978 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.419015 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:04.419044 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.424729 5328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43171
I20260928 08:59:04.424793 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43171 every 8 connection(s)
I20260928 08:59:04.425133 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 08:59:04.427866 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.425861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47125 (local address 127.0.98.62:39955)
0928 08:59:04.425937 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.425939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.426144 (+ 205us) server_negotiation.cc:407] Connection header received
0928 08:59:04.426564 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.426566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.426570 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.426629 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.427676 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.427677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.427677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.427681 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.427694 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.427697 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.427744 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.427801 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.427814 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:59:04.428059 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.425843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.98.3:47125)
0928 08:59:04.426018 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.426055 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.426485 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.426678 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.426681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.426689 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.427538 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.427539 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.427566 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.427594 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.427741 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.427744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.427746 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.427805 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.427852 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260928 08:59:04.428514 5456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39955
I20260928 08:59:04.428602 5456 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:04.428768 5456 heartbeater.cc:511] Master 127.0.98.62:39955 requested a full tablet report, sending...
I20260928 08:59:04.429139 5006 ts_manager.cc:194] Registered new tserver with Master: dffe0b77baf84806ae2ccdb4998d401e (127.0.98.3:43171)
I20260928 08:59:04.432509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5328
I20260928 08:59:04.432595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:04.432830 392 external_mini_cluster.cc:1428] Setting key 17e1e3ac09ad8723770b4a1dbe14e4c2
I20260928 08:59:04.433720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:59:04.435997 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.435019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.0.98.62:39955)
0928 08:59:04.435082 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.435084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.435202 (+ 118us) server_negotiation.cc:407] Connection header received
0928 08:59:04.435250 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.435250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.435255 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.435315 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.435831 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.435832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.435832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.435836 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.435851 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.435854 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.435899 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.435929 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.435941 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:59:04.437718 5006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:59:04.439232 5006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:04.442996 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.441454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.0.1:34342)
0928 08:59:04.441605 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.441649 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.441711 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.442072 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.442077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.442084 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.442563 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.442564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.442581 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.442603 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.442842 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.442847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.442850 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.442909 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.442925 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 08:59:04.443069 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.441551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.98.2:42011)
0928 08:59:04.441721 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.441726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.441738 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:59:04.441766 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.441768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.441779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.441869 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.442691 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.442698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.442698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.442702 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.442739 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.442760 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.442816 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.442957 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.442975 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 08:59:04.443269 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.441682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43713 (local address 127.0.0.1:34808)
0928 08:59:04.441782 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.441792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.441835 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.442360 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.442360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.442363 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.442754 (+ 391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.442755 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.442762 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.442775 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.443142 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.443143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.443143 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.443195 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.443208 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260928 08:59:04.443317 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.442032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.0.98.1:43713)
0928 08:59:04.442238 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.442242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.442253 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:59:04.442278 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.442281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.442289 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.442359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.442940 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.442950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.442950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.442955 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.443038 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.443065 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.443138 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.443215 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.443230 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260928 08:59:04.443799 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 4195bcdf386c4d6a898ea91f8a60777c (DEFAULT_TABLE table=test-table [id=e5c1501172da4d069be2121ab3be952c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:04.443960 5128 tablet_service.cc:1467] Processing CreateTablet for tablet 4195bcdf386c4d6a898ea91f8a60777c (DEFAULT_TABLE table=test-table [id=e5c1501172da4d069be2121ab3be952c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:04.444196 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195bcdf386c4d6a898ea91f8a60777c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:04.444342 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195bcdf386c4d6a898ea91f8a60777c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:04.444651 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.441830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43171 (local address 127.0.0.1:46264)
0928 08:59:04.441973 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.441983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.442022 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.442944 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.442945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.442947 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.444116 (+ 1169us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.444117 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.444123 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.444135 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.444513 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.444514 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.444515 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.444557 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.444569 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260928 08:59:04.444880 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.441954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.0.98.3:43171)
0928 08:59:04.442710 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.442713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.442723 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:04.442750 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.442752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.442761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.442836 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.444284 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.444293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.444293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.444298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.444336 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.444362 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.444494 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.444785 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.444802 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":90,"threads_started":1}
I20260928 08:59:04.445726 5390 tablet_service.cc:1467] Processing CreateTablet for tablet 4195bcdf386c4d6a898ea91f8a60777c (DEFAULT_TABLE table=test-table [id=e5c1501172da4d069be2121ab3be952c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:04.446061 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195bcdf386c4d6a898ea91f8a60777c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:04.447067 5470 tablet_bootstrap.cc:492] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Bootstrap starting.
I20260928 08:59:04.447655 5471 tablet_bootstrap.cc:492] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85: Bootstrap starting.
I20260928 08:59:04.447696 5470 tablet_bootstrap.cc:654] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:04.448014 5470 log.cc:824] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:04.448314 5471 tablet_bootstrap.cc:654] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:04.448587 5471 log.cc:824] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:04.448840 5470 tablet_bootstrap.cc:492] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: No bootstrap required, opened a new log
I20260928 08:59:04.448899 5470 ts_tablet_manager.cc:1397] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:04.449311 5474 tablet_bootstrap.cc:492] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Bootstrap starting.
I20260928 08:59:04.449359 5471 tablet_bootstrap.cc:492] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85: No bootstrap required, opened a new log
I20260928 08:59:04.449424 5471 ts_tablet_manager.cc:1397] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:59:04.449849 5474 tablet_bootstrap.cc:654] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:04.450263 5474 log.cc:824] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:04.450919 5474 tablet_bootstrap.cc:492] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: No bootstrap required, opened a new log
I20260928 08:59:04.451007 5474 ts_tablet_manager.cc:1397] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:04.451790 5470 raft_consensus.cc:348] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.451964 5470 raft_consensus.cc:374] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:04.452000 5470 raft_consensus.cc:729] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434bd76e42134ef9ad1014f9f1c0e20d, State: Initialized, Role: FOLLOWER
I20260928 08:59:04.452037 5471 raft_consensus.cc:348] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.452183 5471 raft_consensus.cc:374] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:04.452143 5470 consensus_queue.cc:260] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.452220 5471 raft_consensus.cc:729] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ddb1e0f8ff4725b3f0dfd7e4583f85, State: Initialized, Role: FOLLOWER
I20260928 08:59:04.452329 5471 consensus_queue.cc:260] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.452519 5325 heartbeater.cc:503] Master 127.0.98.62:39955 was elected leader, sending a full tablet report...
I20260928 08:59:04.452564 5471 ts_tablet_manager.cc:1428] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 08:59:04.452692 5194 heartbeater.cc:503] Master 127.0.98.62:39955 was elected leader, sending a full tablet report...
I20260928 08:59:04.452458 5470 ts_tablet_manager.cc:1428] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:59:04.453239 5474 raft_consensus.cc:348] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.453394 5474 raft_consensus.cc:374] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:04.453439 5474 raft_consensus.cc:729] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffe0b77baf84806ae2ccdb4998d401e, State: Initialized, Role: FOLLOWER
I20260928 08:59:04.453557 5474 consensus_queue.cc:260] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.453827 5474 ts_tablet_manager.cc:1428] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:59:04.453912 5456 heartbeater.cc:503] Master 127.0.98.62:39955 was elected leader, sending a full tablet report...
I20260928 08:59:04.472723 5477 raft_consensus.cc:482] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:04.472838 5477 raft_consensus.cc:504] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.473152 5477 leader_election.cc:288] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 434bd76e42134ef9ad1014f9f1c0e20d (127.0.98.2:42011), dffe0b77baf84806ae2ccdb4998d401e (127.0.98.3:43171)
I20260928 08:59:04.474483 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.473362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.98.1:39799)
0928 08:59:04.473443 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.473458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.473515 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.473667 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.473668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.473671 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.474121 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.474122 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.474129 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.474139 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.474330 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.474331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.474332 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.474381 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.474394 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 08:59:04.474491 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.473407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39799 (local address 127.0.98.2:42011)
0928 08:59:04.473466 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.473469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.473500 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:59:04.473594 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.473595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.473601 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.473654 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.474239 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.474241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.474241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.474246 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.474262 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.474266 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.474324 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.474382 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.474403 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:59:04.474583 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.473576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54669 (local address 127.0.98.3:43171)
0928 08:59:04.473621 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.473623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.473712 (+ 89us) server_negotiation.cc:407] Connection header received
0928 08:59:04.473763 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.473764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.473772 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.473845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.474347 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.474348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.474349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.474353 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.474369 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.474374 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.474430 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.474497 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.474510 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:59:04.474638 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.473478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43171 (local address 127.0.98.1:54669)
0928 08:59:04.473640 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.473651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.473706 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.473851 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.473852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.473853 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:04.474253 (+ 400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:59:04.474253 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:04.474257 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:59:04.474266 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:04.474436 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:04.474436 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:04.474437 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:59:04.474492 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:59:04.474504 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260928 08:59:04.474814 5279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195bcdf386c4d6a898ea91f8a60777c" candidate_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" is_pre_election: true
I20260928 08:59:04.474977 5279 raft_consensus.cc:2454] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8ddb1e0f8ff4725b3f0dfd7e4583f85 in term 0.
I20260928 08:59:04.475243 5082 leader_election.cc:302] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434bd76e42134ef9ad1014f9f1c0e20d, b8ddb1e0f8ff4725b3f0dfd7e4583f85; no voters:
I20260928 08:59:04.475382 5477 raft_consensus.cc:2781] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:04.475447 5477 raft_consensus.cc:482] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:04.475474 5477 raft_consensus.cc:3037] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:04.475386 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195bcdf386c4d6a898ea91f8a60777c" candidate_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffe0b77baf84806ae2ccdb4998d401e" is_pre_election: true
I20260928 08:59:04.475566 5410 raft_consensus.cc:2454] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8ddb1e0f8ff4725b3f0dfd7e4583f85 in term 0.
I20260928 08:59:04.476295 5477 raft_consensus.cc:504] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.476439 5477 leader_election.cc:288] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [CANDIDATE]: Term 1 election: Requested vote from peers 434bd76e42134ef9ad1014f9f1c0e20d (127.0.98.2:42011), dffe0b77baf84806ae2ccdb4998d401e (127.0.98.3:43171)
I20260928 08:59:04.476586 5279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195bcdf386c4d6a898ea91f8a60777c" candidate_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434bd76e42134ef9ad1014f9f1c0e20d"
I20260928 08:59:04.476606 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195bcdf386c4d6a898ea91f8a60777c" candidate_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dffe0b77baf84806ae2ccdb4998d401e"
I20260928 08:59:04.476693 5410 raft_consensus.cc:3037] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:04.476693 5279 raft_consensus.cc:3037] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:04.477586 5410 raft_consensus.cc:2454] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8ddb1e0f8ff4725b3f0dfd7e4583f85 in term 1.
I20260928 08:59:04.477586 5279 raft_consensus.cc:2454] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8ddb1e0f8ff4725b3f0dfd7e4583f85 in term 1.
I20260928 08:59:04.477756 5082 leader_election.cc:302] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 434bd76e42134ef9ad1014f9f1c0e20d, b8ddb1e0f8ff4725b3f0dfd7e4583f85; no voters:
I20260928 08:59:04.477847 5477 raft_consensus.cc:2781] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:04.478008 5477 raft_consensus.cc:686] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [term 1 LEADER]: Becoming Leader. State: Replica: b8ddb1e0f8ff4725b3f0dfd7e4583f85, State: Running, Role: LEADER
I20260928 08:59:04.478080 5477 consensus_queue.cc:237] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } }
I20260928 08:59:04.478895 5004 catalog_manager.cc:5495] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8ddb1e0f8ff4725b3f0dfd7e4583f85 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8ddb1e0f8ff4725b3f0dfd7e4583f85" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:04.483122 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.482187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.0.98.1:43713)
0928 08:59:04.482232 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.482233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.482241 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:04.482306 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.482307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.482318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.482364 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.482918 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.482919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.482919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.482922 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.482931 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.482935 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.482985 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.483019 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.483028 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:59:04.484798 5410 raft_consensus.cc:1261] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 1 FOLLOWER]: Refusing update from remote peer b8ddb1e0f8ff4725b3f0dfd7e4583f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:04.484813 5279 raft_consensus.cc:1261] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 1 FOLLOWER]: Refusing update from remote peer b8ddb1e0f8ff4725b3f0dfd7e4583f85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:04.485152 5477 consensus_queue.cc:1035] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dffe0b77baf84806ae2ccdb4998d401e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:04.485276 5480 consensus_queue.cc:1035] T 4195bcdf386c4d6a898ea91f8a60777c P b8ddb1e0f8ff4725b3f0dfd7e4583f85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "434bd76e42134ef9ad1014f9f1c0e20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:04.487542 5483 mvcc.cc:205] Tried to move back new op lower bound from 7334240028606898176 to 7334240028582686720. Current Snapshot: MvccSnapshot[applied={T|T < 7334240028606898176}]
I20260928 08:59:04.487751 5485 mvcc.cc:205] Tried to move back new op lower bound from 7334240028606898176 to 7334240028582686720. Current Snapshot: MvccSnapshot[applied={T|T < 7334240028606898176}]
I20260928 08:59:04.487867 5484 mvcc.cc:205] Tried to move back new op lower bound from 7334240028606898176 to 7334240028582686720. Current Snapshot: MvccSnapshot[applied={T|T < 7334240028606898176}]
I20260928 08:59:04.490072 5004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:04.490170 5004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:04.490921 5004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ca8b5d0170744f3af279f26c7d0d890: Sending DeleteTablet for 3 replicas of tablet 4195bcdf386c4d6a898ea91f8a60777c
I20260928 08:59:04.491279 5259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195bcdf386c4d6a898ea91f8a60777c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:04 UTC) from {username='slave'} at 127.0.0.1:34342
I20260928 08:59:04.491406 5390 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195bcdf386c4d6a898ea91f8a60777c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:04 UTC) from {username='slave'} at 127.0.0.1:46264
I20260928 08:59:04.491492 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195bcdf386c4d6a898ea91f8a60777c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:04 UTC) from {username='slave'} at 127.0.0.1:34808
I20260928 08:59:04.491521 5493 tablet_replica.cc:330] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: stopping tablet replica
I20260928 08:59:04.491616 5493 raft_consensus.cc:2229] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:04.491712 5493 raft_consensus.cc:2258] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:04.491830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5066
I20260928 08:59:04.492709 5494 tablet_replica.cc:330] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: stopping tablet replica
I20260928 08:59:04.492861 5494 raft_consensus.cc:2229] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:04.492908 5493 ts_tablet_manager.cc:1905] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:04.493078 5494 raft_consensus.cc:2258] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:04.493459 5494 ts_tablet_manager.cc:1905] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:04.494899 5494 ts_tablet_manager.cc:1918] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:04.494902 5493 ts_tablet_manager.cc:1918] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:04.494966 5493 log.cc:1196] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/4195bcdf386c4d6a898ea91f8a60777c
I20260928 08:59:04.494983 5494 log.cc:1196] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/4195bcdf386c4d6a898ea91f8a60777c
I20260928 08:59:04.495235 5494 ts_tablet_manager.cc:1939] T 4195bcdf386c4d6a898ea91f8a60777c P dffe0b77baf84806ae2ccdb4998d401e: Deleting consensus metadata
I20260928 08:59:04.495234 5493 ts_tablet_manager.cc:1939] T 4195bcdf386c4d6a898ea91f8a60777c P 434bd76e42134ef9ad1014f9f1c0e20d: Deleting consensus metadata
I20260928 08:59:04.495671 4991 catalog_manager.cc:4841] TS 434bd76e42134ef9ad1014f9f1c0e20d (127.0.98.2:42011): tablet 4195bcdf386c4d6a898ea91f8a60777c (table test-table [id=e5c1501172da4d069be2121ab3be952c]) successfully deleted
I20260928 08:59:04.495790 4991 catalog_manager.cc:4841] TS dffe0b77baf84806ae2ccdb4998d401e (127.0.98.3:43171): tablet 4195bcdf386c4d6a898ea91f8a60777c (table test-table [id=e5c1501172da4d069be2121ab3be952c]) successfully deleted
W20260928 08:59:04.498025 4991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43713 (error 108)
I20260928 08:59:04.498179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5197
I20260928 08:59:04.498862 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.498736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43713 (local address 127.0.0.1:34830)
0928 08:59:04.498783 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.498820 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 08:59:04.498966 4991 catalog_manager.cc:4584] TS b8ddb1e0f8ff4725b3f0dfd7e4583f85 (127.0.98.1:43713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4195bcdf386c4d6a898ea91f8a60777c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43713: connect: Connection refused (error 111)
I20260928 08:59:04.503046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5328
I20260928 08:59:04.507279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 4975
2026-09-28T08:59:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (604 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 08:59:04.527344 392 mini_oidc.cc:152] Starting JWKS server
I20260928 08:59:04.528385 392 webserver.cc:460] Webserver started at https://127.0.0.1:38895/ using document root <none> and password file <none>
I20260928 08:59:04.528467 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:59:04.528606 392 webserver.cc:460] Webserver started at http://127.0.0.1:40499/ using document root <none> and password file <none>
2026-09-28T08:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:04Z Disabled control of system clock
I20260928 08:59:04.536253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35083
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/testchainca.cert with env {}
W20260928 08:59:04.616498 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:04.616746 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:04.616781 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:04.617950 5502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:04.618016 5502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:04.618040 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:04.618072 5502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:04.618098 5502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:59:04.618129 5502 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:59:04.618156 5502 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38895/jwks.json
I20260928 08:59:04.619661 5502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:04.619933 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:04.620175 5502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:04.625078 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.625105 5502 server_base.cc:1034] running on GCE node
W20260928 08:59:04.625133 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.625294 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.625520 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:04.625816 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:04.627012 5502 hybrid_clock.cc:648] HybridClock initialized: now 1790585944626998 us; error 48 us; skew 500 ppm
I20260928 08:59:04.628252 5502 webserver.cc:460] Webserver started at http://127.0.98.62:38137/ using document root <none> and password file <none>
I20260928 08:59:04.628542 5502 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:04.628615 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:04.628800 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:04.629881 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "3be87ae89317416b90749282928b7827"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "267c3ba185920802111b64b991d81ae4"
server_key_iv: "03b8ab47a1b2eaf40811c34d9d05f2f1"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.630244 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "3be87ae89317416b90749282928b7827"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "267c3ba185920802111b64b991d81ae4"
server_key_iv: "03b8ab47a1b2eaf40811c34d9d05f2f1"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.631589 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:04.632445 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.632771 5502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:04.632876 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "3be87ae89317416b90749282928b7827"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "267c3ba185920802111b64b991d81ae4"
server_key_iv: "03b8ab47a1b2eaf40811c34d9d05f2f1"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.632973 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:04.656509 5502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:04.656903 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:04.657068 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:04.661202 5502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35083
I20260928 08:59:04.661238 5568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35083 every 8 connection(s)
I20260928 08:59:04.661602 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:04.662573 5569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:04.664914 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827: Bootstrap starting.
I20260928 08:59:04.665518 5569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:04.665822 5569 log.cc:824] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:04.666533 5569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827: No bootstrap required, opened a new log
I20260928 08:59:04.668103 5569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be87ae89317416b90749282928b7827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 08:59:04.668243 5569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:04.668267 5569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be87ae89317416b90749282928b7827, State: Initialized, Role: FOLLOWER
I20260928 08:59:04.668421 5569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be87ae89317416b90749282928b7827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 08:59:04.668500 5569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:04.668524 5569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:04.668573 5569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:04.669128 5569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be87ae89317416b90749282928b7827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 08:59:04.669215 5569 leader_election.cc:302] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be87ae89317416b90749282928b7827; no voters:
I20260928 08:59:04.669394 5569 leader_election.cc:288] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:04.669512 5572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:04.669669 5572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [term 1 LEADER]: Becoming Leader. State: Replica: 3be87ae89317416b90749282928b7827, State: Running, Role: LEADER
I20260928 08:59:04.669674 5569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:04.669811 5572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be87ae89317416b90749282928b7827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 08:59:04.670246 5573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be87ae89317416b90749282928b7827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be87ae89317416b90749282928b7827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 08:59:04.670334 5573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:04.670348 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be87ae89317416b90749282928b7827. Latest consensus state: current_term: 1 leader_uuid: "3be87ae89317416b90749282928b7827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be87ae89317416b90749282928b7827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 08:59:04.670614 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:04.670622 5581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:04.670979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5502
I20260928 08:59:04.671089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:04.671160 5581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:04.671300 392 external_mini_cluster.cc:1428] Setting key 0c56118bafb822283b314e93bbf230ce
I20260928 08:59:04.673837 5581 catalog_manager.cc:1347] Generated new cluster ID: 8ac56deed3c94d4c8409fd08cfdc61fc
I20260928 08:59:04.673898 5581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:04.676704 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.671747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.0.98.62:35083)
0928 08:59:04.672118 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.672125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.672148 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:59:04.672239 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.672244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.672295 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.672433 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.672606 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.673759 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.674647 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.675104 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.675248 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:04.675975 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.675979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.675979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.676167 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.676198 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.676209 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.676283 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.676371 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.676450 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":72,"threads_started":1}
I20260928 08:59:04.677614 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:59:04.683357 5581 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:04.683871 5581 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:04.691422 5581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be87ae89317416b90749282928b7827: Generated new TSK 0
I20260928 08:59:04.691612 5581 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:59:04.715193 5599 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 08:59:04.715310 5599 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:59:04.715579 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.698451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.0.98.62:35083)
0928 08:59:04.698518 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.698520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.698658 (+ 138us) server_negotiation.cc:407] Connection header received
0928 08:59:04.698782 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.698782 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.698792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.698853 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:59:04.699018 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.699480 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.700711 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.700905 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.700960 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:04.701212 (+ 252us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:59:04.715335 (+ 14123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:59:04.715517 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 08:59:04.715652 5591 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 08:59:04.716118 5598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:04.698395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.0.1:58078)
0928 08:59:04.698589 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.698602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.698685 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.698881 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.698882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.698885 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:59:04.698947 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:04.698950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.699561 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.699562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:04.700567 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:04.700570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.700986 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.700990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:04.701052 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:04.701073 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:59:04.715536 (+ 14463us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:59:04.715676 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 08:59:04.716789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5502
2026-09-28T08:59:04Z chronyd exiting
I20260928 08:59:04.752758 392 test_util.cc:175] -----------------------------------------------
I20260928 08:59:04.752838 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585920139418-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 08:59:04.753638 392 mini_oidc.cc:152] Starting JWKS server
I20260928 08:59:04.754330 392 webserver.cc:460] Webserver started at https://127.0.0.1:44981/ using document root <none> and password file <none>
I20260928 08:59:04.754374 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:59:04.754544 392 webserver.cc:460] Webserver started at http://127.0.0.1:43143/ using document root <none> and password file <none>
2026-09-28T08:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:04Z Disabled control of system clock
I20260928 08:59:04.761971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/testchainca.cert with env {}
W20260928 08:59:04.842710 5605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:04.842957 5605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:04.842993 5605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:04.844303 5605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:04.844372 5605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:04.844394 5605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:04.844422 5605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:04.844463 5605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:59:04.844489 5605 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:59:04.844514 5605 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44981/jwks.json
I20260928 08:59:04.846103 5605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:04.846385 5605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:04.846614 5605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:59:04.851585 5605 server_base.cc:1034] running on GCE node
W20260928 08:59:04.851572 5611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.851536 5610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:04.851536 5613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:04.851967 5605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:04.852259 5605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:04.853427 5605 hybrid_clock.cc:648] HybridClock initialized: now 1790585944853409 us; error 51 us; skew 500 ppm
I20260928 08:59:04.854755 5605 webserver.cc:460] Webserver started at http://127.0.98.62:33949/ using document root <none> and password file <none>
I20260928 08:59:04.855005 5605 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:04.855073 5605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:04.855228 5605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:04.856343 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "b6ceb9ec98424224ae265012f257ffc8"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "78bc7c0bcea2fe4b87342843e567b855"
server_key_iv: "6f6cd8ed7e91594f0a88ea20bb64a168"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.856774 5605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6ceb9ec98424224ae265012f257ffc8"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "78bc7c0bcea2fe4b87342843e567b855"
server_key_iv: "6f6cd8ed7e91594f0a88ea20bb64a168"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.858073 5605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:04.858865 5619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:04.859040 5605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:04.859138 5605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "b6ceb9ec98424224ae265012f257ffc8"
format_stamp: "Formatted at 2026-09-28 08:59:04 on dist-test-slave-286n"
server_key: "78bc7c0bcea2fe4b87342843e567b855"
server_key_iv: "6f6cd8ed7e91594f0a88ea20bb64a168"
server_key_version: "encryptionkey@0"
I20260928 08:59:04.859236 5605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:04.865960 5605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:04.866251 5605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:04.866400 5605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:04.870301 5605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36743
I20260928 08:59:04.870329 5671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36743 every 8 connection(s)
I20260928 08:59:04.870669 5605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:04.871582 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:04.873852 5672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8: Bootstrap starting.
I20260928 08:59:04.874369 5672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:04.874595 5672 log.cc:824] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:04.875221 5672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8: No bootstrap required, opened a new log
I20260928 08:59:04.875797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5605
I20260928 08:59:04.875908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:04.876120 392 external_mini_cluster.cc:1428] Setting key 52965621e488d461ad1e0269cf4d927f
I20260928 08:59:04.877380 5672 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ceb9ec98424224ae265012f257ffc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260928 08:59:04.877556 5672 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:04.877594 5672 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ceb9ec98424224ae265012f257ffc8, State: Initialized, Role: FOLLOWER
I20260928 08:59:04.877753 5672 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ceb9ec98424224ae265012f257ffc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260928 08:59:04.877846 5672 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:04.877871 5672 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:04.877940 5672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:04.878481 5672 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ceb9ec98424224ae265012f257ffc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260928 08:59:04.878564 5672 leader_election.cc:302] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ceb9ec98424224ae265012f257ffc8; no voters:
I20260928 08:59:04.878728 5672 leader_election.cc:288] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:04.878773 5677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:04.878901 5677 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [term 1 LEADER]: Becoming Leader. State: Replica: b6ceb9ec98424224ae265012f257ffc8, State: Running, Role: LEADER
I20260928 08:59:04.879007 5672 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:04.878993 5677 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ceb9ec98424224ae265012f257ffc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260928 08:59:04.879316 5679 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ceb9ec98424224ae265012f257ffc8. Latest consensus state: current_term: 1 leader_uuid: "b6ceb9ec98424224ae265012f257ffc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ceb9ec98424224ae265012f257ffc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } } }
I20260928 08:59:04.879318 5678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ceb9ec98424224ae265012f257ffc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ceb9ec98424224ae265012f257ffc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } } }
I20260928 08:59:04.879457 5678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:04.879650 5679 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:04.880056 5682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:04.880525 5682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:04.882112 5682 catalog_manager.cc:1347] Generated new cluster ID: 1789ff1c82bc4dd4a479cb1cadbdc7fb
I20260928 08:59:04.882165 5682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:04.882820 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.876628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.98.62:36743)
0928 08:59:04.876826 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.876833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.876856 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:59:04.877360 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.877366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.877422 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.877570 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:04.877778 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.879351 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.880585 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.881284 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.881400 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:04.882158 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:04.882162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:04.882162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:04.882351 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:04.882384 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:04.882394 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:04.882489 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:04.882563 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:59:04.882639 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260928 08:59:04.883790 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:59:04.902827 5682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:04.903537 5682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:04.907891 5682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ceb9ec98424224ae265012f257ffc8: Generated new TSK 0
I20260928 08:59:04.908073 5682 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:59:04.928356 5702 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 08:59:04.928455 5702 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:59:04.928864 5676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:04.914169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.0.98.62:36743)
0928 08:59:04.914217 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:04.914219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:04.914304 (+ 85us) server_negotiation.cc:407] Connection header received
0928 08:59:04.914360 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:04.914361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:04.914372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:04.914439 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:59:04.914584 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.915076 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.916244 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.916509 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.916576 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:04.916818 (+ 242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:59:04.928606 (+ 11788us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:59:04.928803 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 08:59:04.928936 5676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 08:59:04.928993 5701 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:04.914028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36743 (local address 127.0.0.1:52750)
0928 08:59:04.914221 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:04.914232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:04.914299 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:04.914452 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:04.914452 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:04.914454 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:59:04.914515 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:04.914517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.915158 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.915159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:04.916137 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:04.916140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:04.916588 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:04.916592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:04.916680 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:04.916686 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:59:04.928761 (+ 12075us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:59:04.928888 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":133,"threads_started":1}
I20260928 08:59:04.929278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5605
2026-09-28T08:59:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 08:59:04.967005 392 mini_oidc.cc:152] Starting JWKS server
I20260928 08:59:04.967769 392 webserver.cc:460] Webserver started at https://127.0.0.1:36347/ using document root <none> and password file <none>
I20260928 08:59:04.967844 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:59:04.968067 392 webserver.cc:460] Webserver started at http://127.0.0.1:36745/ using document root <none> and password file <none>
2026-09-28T08:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:04Z Disabled control of system clock
I20260928 08:59:04.975777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43761
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43761
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36347/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:59:05.056679 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:05.056886 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:05.056911 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:05.058007 5708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:05.058048 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:05.058063 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:05.058079 5708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:05.058092 5708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:59:05.058107 5708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:59:05.058120 5708 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36347/jwks.json
I20260928 08:59:05.059432 5708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36347/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:05.059630 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:05.059867 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:05.064980 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:05.065011 5708 server_base.cc:1034] running on GCE node
W20260928 08:59:05.064953 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:05.064957 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:05.065466 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:05.065704 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:05.066854 5708 hybrid_clock.cc:648] HybridClock initialized: now 1790585945066839 us; error 29 us; skew 500 ppm
I20260928 08:59:05.068202 5708 webserver.cc:460] Webserver started at http://127.0.98.62:42157/ using document root <none> and password file <none>
I20260928 08:59:05.068529 5708 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:05.068578 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:05.068737 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:05.069881 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "50f047ac423b4b78848693eec3b2bf88"
format_stamp: "Formatted at 2026-09-28 08:59:05 on dist-test-slave-286n"
server_key: "2bf7225801a4dbfb30dedc8d0361df20"
server_key_iv: "72e7cadf3b8423cfad77f423fde0d1a1"
server_key_version: "encryptionkey@0"
I20260928 08:59:05.070292 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "50f047ac423b4b78848693eec3b2bf88"
format_stamp: "Formatted at 2026-09-28 08:59:05 on dist-test-slave-286n"
server_key: "2bf7225801a4dbfb30dedc8d0361df20"
server_key_iv: "72e7cadf3b8423cfad77f423fde0d1a1"
server_key_version: "encryptionkey@0"
I20260928 08:59:05.071611 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:05.072356 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:05.072552 5708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:05.072667 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "50f047ac423b4b78848693eec3b2bf88"
format_stamp: "Formatted at 2026-09-28 08:59:05 on dist-test-slave-286n"
server_key: "2bf7225801a4dbfb30dedc8d0361df20"
server_key_iv: "72e7cadf3b8423cfad77f423fde0d1a1"
server_key_version: "encryptionkey@0"
I20260928 08:59:05.072757 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:05.086599 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:05.087024 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:05.087198 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:05.091411 5708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43761
I20260928 08:59:05.091498 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43761 every 8 connection(s)
I20260928 08:59:05.091858 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:05.092545 5775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:05.095145 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88: Bootstrap starting.
I20260928 08:59:05.095719 5775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:05.096015 5775 log.cc:824] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:05.096819 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88: No bootstrap required, opened a new log
I20260928 08:59:05.098356 5775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f047ac423b4b78848693eec3b2bf88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260928 08:59:05.098492 5775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:05.098515 5775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f047ac423b4b78848693eec3b2bf88, State: Initialized, Role: FOLLOWER
I20260928 08:59:05.098675 5775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f047ac423b4b78848693eec3b2bf88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260928 08:59:05.098747 5775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:05.098780 5775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:05.098831 5775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:05.099315 5775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f047ac423b4b78848693eec3b2bf88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260928 08:59:05.099400 5775 leader_election.cc:302] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50f047ac423b4b78848693eec3b2bf88; no voters:
I20260928 08:59:05.099542 5775 leader_election.cc:288] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:05.099611 5778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:05.099787 5778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [term 1 LEADER]: Becoming Leader. State: Replica: 50f047ac423b4b78848693eec3b2bf88, State: Running, Role: LEADER
I20260928 08:59:05.099807 5775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:05.099892 5778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f047ac423b4b78848693eec3b2bf88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } }
I20260928 08:59:05.099967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5708
I20260928 08:59:05.100085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:05.100260 392 external_mini_cluster.cc:1428] Setting key 01dd08722b8ef1d11af4f6a7294bf50a
I20260928 08:59:05.100236 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f047ac423b4b78848693eec3b2bf88. Latest consensus state: current_term: 1 leader_uuid: "50f047ac423b4b78848693eec3b2bf88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f047ac423b4b78848693eec3b2bf88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } } }
I20260928 08:59:05.100306 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:05.100519 5784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:05.100524 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50f047ac423b4b78848693eec3b2bf88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f047ac423b4b78848693eec3b2bf88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43761 } } }
I20260928 08:59:05.101202 5784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:05.101199 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:05.102813 5784 catalog_manager.cc:1347] Generated new cluster ID: 97105d715e494ccdba8cbaa0ed6bf986
I20260928 08:59:05.102862 5784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:05.106657 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:05.100907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.0.98.62:43761)
0928 08:59:05.101224 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:05.101234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:05.101266 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:59:05.101384 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:05.101392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:05.101462 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:05.101993 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:05.102096 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.103428 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.104495 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.105142 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.105267 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:05.105950 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:05.105954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:05.105954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:05.106159 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:05.106200 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:05.106219 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:05.106290 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:05.106357 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:59:05.106432 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":64,"threads_started":1}
I20260928 08:59:05.107565 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:59:05.115082 5784 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:05.115731 5784 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:05.125432 5784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50f047ac423b4b78848693eec3b2bf88: Generated new TSK 0
I20260928 08:59:05.125613 5784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:59:05.138744 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:05.134992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.98.62:43761)
0928 08:59:05.135050 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:05.135052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:05.135150 (+ 98us) server_negotiation.cc:407] Connection header received
0928 08:59:05.135215 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:05.135216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:05.135229 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:05.135297 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:59:05.135400 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.136176 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.136995 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.137293 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.137365 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:05.137612 (+ 247us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:59:05.138540 (+ 928us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:59:05.138667 (+ 127us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 08:59:05.138798 5792 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 08:59:05.138863 5804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:05.134898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43761 (local address 127.0.0.1:35290)
0928 08:59:05.135075 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:05.135093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:05.135147 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:05.135304 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:05.135304 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:05.135306 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:59:05.135352 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:05.135353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.136253 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.136254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:05.136913 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:05.136915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.137438 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.137447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:05.137501 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:05.137506 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:59:05.138652 (+ 1146us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:59:05.138758 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43761: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36347/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260928 08:59:05.141005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5708
E20260928 08:59:05.151233 5805 webserver.cc:533] Webserver: sslize failed:
E20260928 08:59:05.151325 5805 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T08:59:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T08:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:05Z Disabled control of system clock
I20260928 08:59:05.186018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35299
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35299
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/testchainca.cert with env {}
W20260928 08:59:05.265914 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:05.266108 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:05.266129 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:05.267302 5809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:05.267345 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:05.267357 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:05.267370 5809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:05.267383 5809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:59:05.267421 5809 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:59:05.267449 5809 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 08:59:05.269055 5809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:05.269349 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:05.269591 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:05.274284 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:05.274297 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:05.274420 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:05.274580 5809 server_base.cc:1034] running on GCE node
I20260928 08:59:05.274837 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:05.275122 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:05.276288 5809 hybrid_clock.cc:648] HybridClock initialized: now 1790585945276281 us; error 36 us; skew 500 ppm
I20260928 08:59:05.277583 5809 webserver.cc:460] Webserver started at http://127.0.98.62:46463/ using document root <none> and password file <none>
I20260928 08:59:05.277822 5809 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:05.277913 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:05.278051 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:05.279065 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "f36f561b66284cbf8062daa4ad3c6b46"
format_stamp: "Formatted at 2026-09-28 08:59:05 on dist-test-slave-286n"
server_key: "72f93dca9ac3216f7709dd5eb5b0ef67"
server_key_iv: "fdef3769d26485f3a116ad53da333eca"
server_key_version: "encryptionkey@0"
I20260928 08:59:05.279443 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "f36f561b66284cbf8062daa4ad3c6b46"
format_stamp: "Formatted at 2026-09-28 08:59:05 on dist-test-slave-286n"
server_key: "72f93dca9ac3216f7709dd5eb5b0ef67"
server_key_iv: "fdef3769d26485f3a116ad53da333eca"
server_key_version: "encryptionkey@0"
I20260928 08:59:05.280865 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:05.281648 5823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:05.281876 5809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:05.282006 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "f36f561b66284cbf8062daa4ad3c6b46"
format_stamp: "Formatted at 2026-09-28 08:59:05 on dist-test-slave-286n"
server_key: "72f93dca9ac3216f7709dd5eb5b0ef67"
server_key_iv: "fdef3769d26485f3a116ad53da333eca"
server_key_version: "encryptionkey@0"
I20260928 08:59:05.282102 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:05.289417 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:05.289706 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:05.289865 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:05.294036 5809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35299
I20260928 08:59:05.294080 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35299 every 8 connection(s)
I20260928 08:59:05.294451 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:05.295356 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:05.297622 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46: Bootstrap starting.
I20260928 08:59:05.298156 5876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:05.298389 5876 log.cc:824] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:05.299029 5876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46: No bootstrap required, opened a new log
I20260928 08:59:05.300143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5809
I20260928 08:59:05.300259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:05.300454 392 external_mini_cluster.cc:1428] Setting key 58d317e0b0e90b455d23f7749f9ac54d
I20260928 08:59:05.301095 5876 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f561b66284cbf8062daa4ad3c6b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35299 } }
I20260928 08:59:05.301296 5876 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:05.301337 5876 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36f561b66284cbf8062daa4ad3c6b46, State: Initialized, Role: FOLLOWER
I20260928 08:59:05.301458 5876 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f561b66284cbf8062daa4ad3c6b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35299 } }
I20260928 08:59:05.301532 5876 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:05.301569 5876 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:05.301622 5876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:05.302356 5876 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f561b66284cbf8062daa4ad3c6b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35299 } }
I20260928 08:59:05.302464 5876 leader_election.cc:302] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36f561b66284cbf8062daa4ad3c6b46; no voters:
I20260928 08:59:05.302610 5876 leader_election.cc:288] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:05.302635 5881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:05.302798 5881 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [term 1 LEADER]: Becoming Leader. State: Replica: f36f561b66284cbf8062daa4ad3c6b46, State: Running, Role: LEADER
I20260928 08:59:05.302865 5876 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:05.302888 5881 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f561b66284cbf8062daa4ad3c6b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35299 } }
I20260928 08:59:05.303181 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36f561b66284cbf8062daa4ad3c6b46. Latest consensus state: current_term: 1 leader_uuid: "f36f561b66284cbf8062daa4ad3c6b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f561b66284cbf8062daa4ad3c6b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35299 } } }
I20260928 08:59:05.303272 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:05.303191 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36f561b66284cbf8062daa4ad3c6b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36f561b66284cbf8062daa4ad3c6b46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35299 } } }
I20260928 08:59:05.303356 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:05.303860 5887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:05.304414 5887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:05.305991 5887 catalog_manager.cc:1347] Generated new cluster ID: 2ce838a37a4d4b43b4e2b3c93f6f8fb0
I20260928 08:59:05.306080 5887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:05.306989 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:05.301015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.0.98.62:35299)
0928 08:59:05.301259 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:05.301268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:05.301302 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:59:05.301416 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:05.301424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:05.301493 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:05.301655 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:05.301838 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.303576 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.304682 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.305310 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.305414 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:05.306280 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:05.306284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:05.306284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:59:05.306484 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:05.306524 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:59:05.306538 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:05.306610 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:05.306702 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:59:05.306781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260928 08:59:05.307864 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:59:05.312798 5887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:05.313265 5887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:05.319428 5887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36f561b66284cbf8062daa4ad3c6b46: Generated new TSK 0
I20260928 08:59:05.319636 5887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:59:05.795473 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:05.323925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.0.98.62:35299)
0928 08:59:05.323971 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:05.323972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:05.324082 (+ 110us) server_negotiation.cc:407] Connection header received
0928 08:59:05.324161 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:05.324162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:05.324181 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:05.324241 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:59:05.324370 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.324890 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.326019 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.326218 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.326266 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:05.326458 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:59:05.795212 (+468754us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:59:05.795413 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44370: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 08:59:05.795549 5880 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44370: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 08:59:05.795656 5905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:05.323881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35299 (local address 127.0.0.1:44370)
0928 08:59:05.324033 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:05.324042 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:05.324098 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:05.324255 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:05.324255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:05.324257 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:59:05.324313 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:05.324315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.324983 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.324984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:05.325927 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:05.325930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:05.326304 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:05.326310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:05.326369 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:05.326374 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:59:05.795347 (+468973us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:59:05.795506 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35299: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260928 08:59:05.796164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5809
2026-09-28T08:59:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (635 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:06 dist-test-slave-286n krb5kdc[5917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:06 dist-test-slave-286n krb5kdc[5917](info): set up 2 sockets
Sep 28 08:59:06 dist-test-slave-286n krb5kdc[5917](info): commencing operation
krb5kdc: starting...
W20260928 08:59:07.029387 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:59:07.148481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:07.230540 5930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:07.230770 5930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:07.230795 5930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:07.231992 5930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:07.232038 5930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:07.232055 5930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:07.232069 5930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:07.232084 5930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:07.233767 5930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:07.234045 5930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:07.234301 5930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:07.239331 5936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.239334 5935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.239367 5938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:07.239789 5930 server_base.cc:1034] running on GCE node
I20260928 08:59:07.240036 5930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:07.240317 5930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:07.241461 5930 hybrid_clock.cc:648] HybridClock initialized: now 1790585947241398 us; error 80 us; skew 500 ppm
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.251154 5930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:59:07.251672 5930 webserver.cc:460] Webserver started at http://127.0.98.62:45589/ using document root <none> and password file <none>
I20260928 08:59:07.251917 5930 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:07.251963 5930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:07.252118 5930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:07.253762 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "8fff18600569407d8a682f8c99d27e23"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "8f19101a1c1d1be60d256644a4d07ff1"
server_key_iv: "b8d105124462f3202fe800980b57a78e"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.254201 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "8fff18600569407d8a682f8c99d27e23"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "8f19101a1c1d1be60d256644a4d07ff1"
server_key_iv: "b8d105124462f3202fe800980b57a78e"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.256054 5930 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:59:07.257153 5945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.257560 5930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 08:59:07.257670 5930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "8fff18600569407d8a682f8c99d27e23"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "8f19101a1c1d1be60d256644a4d07ff1"
server_key_iv: "b8d105124462f3202fe800980b57a78e"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.257742 5930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:07.284040 5930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:07.287206 5930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:07.287381 5930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:07.292922 5930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42313
I20260928 08:59:07.292961 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42313 every 8 connection(s)
I20260928 08:59:07.293648 5930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:07.294369 5998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:07.296849 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23: Bootstrap starting.
I20260928 08:59:07.297448 5998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:07.297801 5998 log.cc:824] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:07.298545 5998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23: No bootstrap required, opened a new log
I20260928 08:59:07.300141 5998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fff18600569407d8a682f8c99d27e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42313 } }
I20260928 08:59:07.300310 5998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:07.300367 5998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fff18600569407d8a682f8c99d27e23, State: Initialized, Role: FOLLOWER
I20260928 08:59:07.300540 5998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fff18600569407d8a682f8c99d27e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42313 } }
I20260928 08:59:07.300674 5998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:07.300721 5998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:07.300776 5998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:07.301666 5998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fff18600569407d8a682f8c99d27e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42313 } }
I20260928 08:59:07.301785 5998 leader_election.cc:302] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fff18600569407d8a682f8c99d27e23; no voters:
I20260928 08:59:07.301959 5998 leader_election.cc:288] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:07.301992 6001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:07.302178 6001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [term 1 LEADER]: Becoming Leader. State: Replica: 8fff18600569407d8a682f8c99d27e23, State: Running, Role: LEADER
I20260928 08:59:07.302240 5998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:07.302260 6001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fff18600569407d8a682f8c99d27e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42313 } }
I20260928 08:59:07.302627 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fff18600569407d8a682f8c99d27e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fff18600569407d8a682f8c99d27e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42313 } } }
I20260928 08:59:07.302639 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fff18600569407d8a682f8c99d27e23. Latest consensus state: current_term: 1 leader_uuid: "8fff18600569407d8a682f8c99d27e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fff18600569407d8a682f8c99d27e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42313 } } }
I20260928 08:59:07.302749 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:07.302753 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:07.303027 6010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:07.303503 6010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:07.303819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 5930
I20260928 08:59:07.303921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:07.304101 392 external_mini_cluster.cc:1428] Setting key a5333a30363731cc270f4c6e8efa55db
I20260928 08:59:07.305258 6010 catalog_manager.cc:1347] Generated new cluster ID: eb262f3b4f7347a09d1cb64c6582623d
I20260928 08:59:07.305318 6010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:07.316684 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.304766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.0.98.62:42313)
0928 08:59:07.304940 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.304950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.305525 (+ 575us) server_negotiation.cc:407] Connection header received
0928 08:59:07.305639 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.305646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.305706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.305832 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.306593 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.307954 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.309041 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.309681 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.309776 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.312867 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.312872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.312872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.312885 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.314935 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.315341 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.315344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.315345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.315459 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.315674 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.315675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.315675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.316012 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.316177 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.316254 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.316410 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260928 08:59:07.335549 6010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:07.336329 6010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:07.348351 6010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fff18600569407d8a682f8c99d27e23: Generated new TSK 0
I20260928 08:59:07.348623 6010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:59:07.385708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42313
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:07.466847 6024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:07.467046 6024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:07.467069 6024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:07.468219 6024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:07.468269 6024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:07.468297 6024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:07.469856 6024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:07.470155 6024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:07.470412 6024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:07.475317 6029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.475319 6030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.475304 6032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:07.475538 6024 server_base.cc:1034] running on GCE node
I20260928 08:59:07.475829 6024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:07.476061 6024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:07.477237 6024 hybrid_clock.cc:648] HybridClock initialized: now 1790585947477217 us; error 36 us; skew 500 ppm
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.480024 6024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:59:07.480486 6024 webserver.cc:460] Webserver started at http://127.0.98.1:39549/ using document root <none> and password file <none>
I20260928 08:59:07.480803 6024 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:07.480880 6024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:07.481060 6024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:07.482209 6024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a9c1200084d45218fca23f674bba8b7"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "e4d9d51198d7e39ceb22d436248084ac"
server_key_iv: "977874247aeb07c2455030970abd6e07"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.482614 6024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a9c1200084d45218fca23f674bba8b7"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "e4d9d51198d7e39ceb22d436248084ac"
server_key_iv: "977874247aeb07c2455030970abd6e07"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.484058 6024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:07.484890 6039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.485059 6024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:07.485173 6024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "5a9c1200084d45218fca23f674bba8b7"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "e4d9d51198d7e39ceb22d436248084ac"
server_key_iv: "977874247aeb07c2455030970abd6e07"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.485265 6024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:07.508504 6024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:07.510815 6024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:07.511020 6024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:07.511287 6024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:07.511646 6024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:07.511706 6024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.511747 6024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:07.511799 6024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.517989 6024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37537
I20260928 08:59:07.518036 6152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37537 every 8 connection(s)
I20260928 08:59:07.518440 6024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:07.520761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6024
I20260928 08:59:07.520881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:07.521076 392 external_mini_cluster.cc:1428] Setting key cef3ff3bb2fdc9b6c108fe1c0eaaae86
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:07.528808 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.519325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46091 (local address 127.0.98.62:42313)
0928 08:59:07.519376 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.519378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.519585 (+ 207us) server_negotiation.cc:407] Connection header received
0928 08:59:07.520541 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.520543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.520566 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.520626 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.521513 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.522008 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.523060 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.523283 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.523360 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.525887 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.525889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.525889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.525897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.527353 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.527773 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.527774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.527774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.527833 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.528085 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.528086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.528086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.528201 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.528305 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.528585 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.528699 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:59:07.529016 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.519249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42313 (local address 127.0.98.1:46091)
0928 08:59:07.519451 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:07.519488 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:07.520427 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:07.520731 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:07.520735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:07.521113 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:07.521431 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:07.521438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.522121 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.522122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:07.522955 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:07.522957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.523366 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.523367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:07.523504 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.524052 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:07.524063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:07.525784 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:07.527439 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:07.527446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:07.527448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:07.527673 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:07.527901 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:07.527902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:07.527902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:07.528016 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:07.528299 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:07.528306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:07.528451 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 08:59:07.528563 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:59:07.528737 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 08:59:07.529512 6153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42313
I20260928 08:59:07.529604 6153 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:07.529752 6153 heartbeater.cc:511] Master 127.0.98.62:42313 requested a full tablet report, sending...
I20260928 08:59:07.530277 5962 ts_manager.cc:194] Registered new tserver with Master: 5a9c1200084d45218fca23f674bba8b7 (127.0.98.1:37537)
I20260928 08:59:07.530936 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:59:07.548449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42313
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:07.628168 6158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:07.628403 6158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:07.628443 6158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:07.629662 6158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:07.629734 6158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:07.629781 6158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:07.631242 6158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:07.631506 6158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:07.631733 6158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:07.636891 6163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.636873 6166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.636873 6164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:07.636897 6158 server_base.cc:1034] running on GCE node
I20260928 08:59:07.637360 6158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:07.637591 6158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:07.638753 6158 hybrid_clock.cc:648] HybridClock initialized: now 1790585947638737 us; error 29 us; skew 500 ppm
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.641419 6158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:59:07.641949 6158 webserver.cc:460] Webserver started at http://127.0.98.2:46139/ using document root <none> and password file <none>
I20260928 08:59:07.642190 6158 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:07.642239 6158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:07.642405 6158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:07.643508 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "aafbe36d0b3148419c239e82b05fa084"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "6cc7c17d4cdd7bc9cd0fa787ff64349e"
server_key_iv: "15ef9661cc22ef34dc736604495b9473"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.643877 6158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aafbe36d0b3148419c239e82b05fa084"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "6cc7c17d4cdd7bc9cd0fa787ff64349e"
server_key_iv: "15ef9661cc22ef34dc736604495b9473"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.645364 6158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 08:59:07.646178 6173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.646358 6158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:07.646457 6158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "aafbe36d0b3148419c239e82b05fa084"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "6cc7c17d4cdd7bc9cd0fa787ff64349e"
server_key_iv: "15ef9661cc22ef34dc736604495b9473"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.646528 6158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:07.657133 6158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:07.659145 6158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:07.659296 6158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:07.659513 6158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:07.659803 6158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:07.659865 6158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.659909 6158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:07.659934 6158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.665858 6158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43403
I20260928 08:59:07.665894 6286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43403 every 8 connection(s)
I20260928 08:59:07.666196 6158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:07.673295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6158
I20260928 08:59:07.673415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:07.673636 392 external_mini_cluster.cc:1428] Setting key 46edeb5766f751e3e7258dadd54e1eb4
I20260928 08:59:07.676590 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.666970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58949 (local address 127.0.98.62:42313)
0928 08:59:07.667079 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.667081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.667269 (+ 188us) server_negotiation.cc:407] Connection header received
0928 08:59:07.668302 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.668304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.668319 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.668389 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.669204 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.669711 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.670815 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.671017 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.671111 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.673409 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.673411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.673411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.673420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.675039 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.675463 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.675465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.675465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.675526 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.675878 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.675879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.675879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.676025 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.676093 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.676355 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.676460 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:59:07.676606 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.666893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42313 (local address 127.0.98.2:58949)
0928 08:59:07.667125 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:07.667176 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:07.668175 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:07.668471 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:07.668475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:07.668878 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:07.669085 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:07.669090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.669825 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.669826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:07.670699 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:07.670701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.671133 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.671134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:07.671264 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.671806 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:07.671814 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:07.673290 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:07.675143 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:07.675150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:07.675152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:07.675374 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:07.675623 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:07.675625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:07.675625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:07.675786 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:07.676091 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:07.676096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:07.676226 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 08:59:07.676317 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:59:07.676449 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260928 08:59:07.677131 6287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42313
I20260928 08:59:07.677248 6287 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:07.677424 6287 heartbeater.cc:511] Master 127.0.98.62:42313 requested a full tablet report, sending...
I20260928 08:59:07.677776 5962 ts_manager.cc:194] Registered new tserver with Master: aafbe36d0b3148419c239e82b05fa084 (127.0.98.2:43403)
I20260928 08:59:07.678315 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58949
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:59:07.700181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42313
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:07.779816 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:07.780128 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:07.780164 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:07.781867 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:07.781932 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:07.781977 6292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:07.783794 6292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:07.784039 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:07.784286 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:07.789770 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.789775 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:07.789793 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:07.790104 6292 server_base.cc:1034] running on GCE node
I20260928 08:59:07.790313 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:07.790545 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:07.791709 6292 hybrid_clock.cc:648] HybridClock initialized: now 1790585947791699 us; error 28 us; skew 500 ppm
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.794153 6292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:59:07.794636 6292 webserver.cc:460] Webserver started at http://127.0.98.3:45087/ using document root <none> and password file <none>
I20260928 08:59:07.794870 6292 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:07.794939 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:07.795094 6292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:07.796123 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "458d6c42040c4044bdef980746a43770"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "5ca85075985e3029811359c2cbcd79ab"
server_key_iv: "6b7eb569016be139460408108e483bf2"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.796499 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "458d6c42040c4044bdef980746a43770"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "5ca85075985e3029811359c2cbcd79ab"
server_key_iv: "6b7eb569016be139460408108e483bf2"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.797837 6292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:07.798682 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.798897 6292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:07.799003 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "458d6c42040c4044bdef980746a43770"
format_stamp: "Formatted at 2026-09-28 08:59:07 on dist-test-slave-286n"
server_key: "5ca85075985e3029811359c2cbcd79ab"
server_key_iv: "6b7eb569016be139460408108e483bf2"
server_key_version: "encryptionkey@0"
I20260928 08:59:07.799091 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:07.817617 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:07.819602 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:07.819795 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:07.820034 6292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:07.820331 6292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:07.820386 6292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.820428 6292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:07.820456 6292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:07.826445 6292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45337
I20260928 08:59:07.826493 6420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45337 every 8 connection(s)
I20260928 08:59:07.826839 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:07.834921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6292
I20260928 08:59:07.835012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:07.835188 392 external_mini_cluster.cc:1428] Setting key 76827a5fb2741a03ab3973e8e1e75381
I20260928 08:59:07.836503 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.827592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49387 (local address 127.0.98.62:42313)
0928 08:59:07.827658 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.827660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.827888 (+ 228us) server_negotiation.cc:407] Connection header received
0928 08:59:07.828879 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.828880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.828891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.828956 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.829542 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.830083 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.831116 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.831305 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.831360 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.833432 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.833433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.833433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.833439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.834868 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.835331 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.835333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.835333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.835398 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.835732 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.835733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.835733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.835895 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.836000 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.836314 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.836389 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:59:07.836707 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.827586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42313 (local address 127.0.98.3:49387)
0928 08:59:07.827739 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:07.827788 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:07.828750 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:07.829009 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:07.829013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:07.829249 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:07.829434 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:07.829438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.830206 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.830207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:07.831028 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:07.831030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.831380 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.831381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:07.831476 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.831896 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:07.831903 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:07.833345 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:07.834947 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:07.834953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:07.834955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:07.835251 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:07.835466 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:07.835467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:07.835467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:07.835623 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:07.835995 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:07.836001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:07.836178 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 08:59:07.836281 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:59:07.836449 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260928 08:59:07.837141 6421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42313
I20260928 08:59:07.837217 6421 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:07.837406 6421 heartbeater.cc:511] Master 127.0.98.62:42313 requested a full tablet report, sending...
I20260928 08:59:07.837819 5962 ts_manager.cc:194] Registered new tserver with Master: 458d6c42040c4044bdef980746a43770 (127.0.98.3:45337)
I20260928 08:59:07.838152 5962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49387
I20260928 08:59:07.847183 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/test-admin.keytab.
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.863371 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 08:59:07.865024 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:07.864234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.0.98.1:37537)
0928 08:59:07.864394 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.864402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.864416 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:59:07.864439 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.864440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.864453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.864538 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.864935 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35734: BlockingRecv error: recv got EOF from 127.0.0.1:35734 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
W20260928 08:59:07.865109 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.866142 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.867085 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:59:07.874608 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.867573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.0.98.1:37537)
0928 08:59:07.867630 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.867632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.867661 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:59:07.867738 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.867739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.867749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.867797 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.868147 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.868969 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.869684 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.870028 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.870085 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.872098 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.872111 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.872111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.872119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.873629 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.873935 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.873937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.873937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.874014 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.874178 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.874179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.874179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.874306 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.874409 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.874474 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.874539 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 08:59:07.876405 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:07.875849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.0.98.1:37537)
0928 08:59:07.875909 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.875911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.875923 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:59:07.875995 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.875996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.876006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.876060 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.876359 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35754: BlockingRecv error: recv got EOF from 127.0.0.1:35754 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 08:59:07.876537 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc)
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.877494 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.878338 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:59:07.885396 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.878981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.0.98.1:37537)
0928 08:59:07.879040 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.879042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.879108 (+ 66us) server_negotiation.cc:407] Connection header received
0928 08:59:07.879164 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.879165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.879176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.879245 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.879607 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.880062 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.880821 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.881006 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.881090 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.883058 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.883059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.883059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.883065 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.884450 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.884781 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.884781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.884782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.884829 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.884977 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.884978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.884978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.885052 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.885111 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.885195 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.885251 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 08:59:07.886693 6428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:07.886306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35774 (local address 127.0.98.1:37537)
0928 08:59:07.886357 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.886358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.886368 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:07.886392 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.886392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.886399 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.886439 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.886664 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35774: BlockingRecv error: recv got EOF from 127.0.0.1:35774 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 08:59:07.886785 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585920139418-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.887710 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:07.888531 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[5917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 08:59:07.895308 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:07.889114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.0.98.1:37537)
0928 08:59:07.889150 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:07.889152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:07.889190 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:59:07.889260 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:07.889261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:07.889269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:07.889324 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:07.889688 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.890233 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.890922 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:07.891111 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:07.891194 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:07.893072 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:07.893073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:07.893074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:07.893086 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:07.894202 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.894557 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.894558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.894558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.894614 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:07.894830 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:07.894830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:07.894831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:07.894927 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:07.895030 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:07.895129 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 08:59:07.895197 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:07.895965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6024
I20260928 08:59:07.900748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6158
I20260928 08:59:07.905030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6292
I20260928 08:59:07.908974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 5930
2026-09-28T08:59:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1942 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[6441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[6441](info): set up 2 sockets
Sep 28 08:59:07 dist-test-slave-286n krb5kdc[6441](info): commencing operation
krb5kdc: starting...
W20260928 08:59:08.953208 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 08:59:09.072264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:09.151422 6454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:09.151666 6454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:09.151701 6454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:09.152870 6454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:09.152945 6454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:09.152971 6454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:09.152999 6454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:09.153025 6454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:09.154522 6454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:09.154799 6454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:09.155035 6454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:09.159754 6462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.159839 6460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.159809 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.160044 6454 server_base.cc:1034] running on GCE node
I20260928 08:59:09.160259 6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:09.160481 6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:09.161623 6454 hybrid_clock.cc:648] HybridClock initialized: now 1790585949161612 us; error 28 us; skew 500 ppm
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:09.164117 6454 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 08:59:09.164484 6454 webserver.cc:460] Webserver started at http://127.0.98.62:38273/ using document root <none> and password file <none>
I20260928 08:59:09.164740 6454 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.164794 6454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.164891 6454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:09.165885 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "f2dbcf189fc949769c80262bc045854d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "b23892b60138fe0d301ebee829a2e450"
server_key_iv: "84e30cc42980389f2f656a0d2e171d1e"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.166193 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2dbcf189fc949769c80262bc045854d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "b23892b60138fe0d301ebee829a2e450"
server_key_iv: "84e30cc42980389f2f656a0d2e171d1e"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.167483 6454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:09.168201 6469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.168362 6454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:09.168435 6454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "f2dbcf189fc949769c80262bc045854d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "b23892b60138fe0d301ebee829a2e450"
server_key_iv: "84e30cc42980389f2f656a0d2e171d1e"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.168526 6454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.183103 6454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:09.185402 6454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:09.185554 6454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:09.189424 6454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36763
I20260928 08:59:09.189473 6521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36763 every 8 connection(s)
I20260928 08:59:09.189823 6454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:09.190747 6522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:09.192940 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Bootstrap starting.
I20260928 08:59:09.193504 6522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:09.193783 6522 log.cc:824] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:09.194458 6522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: No bootstrap required, opened a new log
I20260928 08:59:09.195958 6522 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260928 08:59:09.196112 6522 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:09.196159 6522 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2dbcf189fc949769c80262bc045854d, State: Initialized, Role: FOLLOWER
I20260928 08:59:09.196321 6522 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260928 08:59:09.196422 6522 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:09.196471 6522 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:09.196519 6522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:09.196770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6454
I20260928 08:59:09.196909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:09.197113 392 external_mini_cluster.cc:1428] Setting key 9812b89c2b12d4271a3494c20388ce7a
I20260928 08:59:09.197584 6522 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260928 08:59:09.197698 6522 leader_election.cc:302] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2dbcf189fc949769c80262bc045854d; no voters:
I20260928 08:59:09.197835 6522 leader_election.cc:288] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:09.197921 6527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:09.198068 6527 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 1 LEADER]: Becoming Leader. State: Replica: f2dbcf189fc949769c80262bc045854d, State: Running, Role: LEADER
I20260928 08:59:09.198066 6522 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:09.198175 6527 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } }
I20260928 08:59:09.199231 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2dbcf189fc949769c80262bc045854d. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } }
I20260928 08:59:09.199327 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:09.199545 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:09.199730 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } }
I20260928 08:59:09.199853 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:09.200263 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:09.201843 6541 catalog_manager.cc:1347] Generated new cluster ID: 3410b440408d4eddb5e3bbad63cce68f
I20260928 08:59:09.201900 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:09.207819 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.197588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.0.98.62:36763)
0928 08:59:09.197850 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.197860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.197902 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:59:09.198022 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.198030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.198090 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.199052 (+ 962us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.199162 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.200669 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.201700 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.202204 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.202345 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.204608 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.204612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.204612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.204623 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.206306 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.206686 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.206688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.206689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.206794 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.207026 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.207027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.207027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.207250 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.207423 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.207494 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.207631 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260928 08:59:09.217245 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:09.217796 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:09.221665 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Generated new TSK 0
I20260928 08:59:09.221846 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 08:59:09.251554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:09.335788 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:09.336086 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:09.336128 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:09.337594 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:09.337668 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:09.337723 6548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:09.339241 6548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:09.339519 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:09.339761 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:09.345299 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.345351 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.345294 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.345611 6548 server_base.cc:1034] running on GCE node
I20260928 08:59:09.345816 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:09.346060 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:09.347234 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790585949347218 us; error 32 us; skew 500 ppm
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:09.349882 6548 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 08:59:09.350322 6548 webserver.cc:460] Webserver started at http://127.0.98.1:38355/ using document root <none> and password file <none>
I20260928 08:59:09.350555 6548 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.350625 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.350775 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:09.351846 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "08db1b76e35e496a98da7649d4a35f0d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "f2fca13932db71b272aa473410df5ee7"
server_key_iv: "4190cbfbfb3f225fc8c7ffe4f99dd4b9"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.352217 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08db1b76e35e496a98da7649d4a35f0d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "f2fca13932db71b272aa473410df5ee7"
server_key_iv: "4190cbfbfb3f225fc8c7ffe4f99dd4b9"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.353717 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:09.354471 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.354662 6548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:09.354759 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "08db1b76e35e496a98da7649d4a35f0d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "f2fca13932db71b272aa473410df5ee7"
server_key_iv: "4190cbfbfb3f225fc8c7ffe4f99dd4b9"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.354838 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.373689 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:09.375530 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:09.375690 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:09.375929 6548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:09.376228 6548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:09.376281 6548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.376314 6548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:09.376341 6548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.381996 6548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35911
I20260928 08:59:09.382037 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35911 every 8 connection(s)
I20260928 08:59:09.382340 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:09.386996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6548
I20260928 08:59:09.387163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:09.387405 392 external_mini_cluster.cc:1428] Setting key d8d68b1318f15b9858806d1e3af574cd
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:09.392292 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.382930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:47827)
0928 08:59:09.383113 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.383155 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.383661 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.383992 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.383997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.384568 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.384768 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.384773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.385531 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.385532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.386323 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.386325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.386740 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.386741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.386880 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.387509 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.387518 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.388876 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.390900 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.390907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.390908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.391141 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.391355 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.391356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.391356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.391450 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.391743 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.391747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.391919 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.392007 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.392133 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260928 08:59:09.392284 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.382958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47827 (local address 127.0.98.62:36763)
0928 08:59:09.383028 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.383030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.383240 (+ 210us) server_negotiation.cc:407] Connection header received
0928 08:59:09.383783 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.383784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.383801 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.383915 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.384887 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.385426 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.386427 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.386643 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.386729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.388978 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.388980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.388980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.388988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.390764 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.391220 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.391221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.391221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.391287 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.391518 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.391519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.391520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.391672 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.391751 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.392056 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.392151 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 08:59:09.392869 6677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260928 08:59:09.392989 6677 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:09.393200 6677 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260928 08:59:09.393688 6486 ts_manager.cc:194] Registered new tserver with Master: 08db1b76e35e496a98da7649d4a35f0d (127.0.98.1:35911)
I20260928 08:59:09.394308 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47827
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 08:59:09.413487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:09.495854 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:09.496063 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:09.496086 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:09.497301 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:09.497342 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:09.497368 6682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:09.498875 6682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:09.499126 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:09.499366 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:09.504294 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.504395 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.504549 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.504688 6682 server_base.cc:1034] running on GCE node
I20260928 08:59:09.504904 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:09.505142 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:09.506286 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790585949506274 us; error 33 us; skew 500 ppm
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:09.508913 6682 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 08:59:09.509420 6682 webserver.cc:460] Webserver started at http://127.0.98.2:39839/ using document root <none> and password file <none>
I20260928 08:59:09.509680 6682 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.509727 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.509869 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:09.510938 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "4908d193378042b4b66566b3d42c99c7"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "1a3e91d34721f90093788105c2a25ec9"
server_key_iv: "46b0872e24e8a97921b1ff513d535826"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.511303 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4908d193378042b4b66566b3d42c99c7"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "1a3e91d34721f90093788105c2a25ec9"
server_key_iv: "46b0872e24e8a97921b1ff513d535826"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.512738 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 08:59:09.513526 6697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.513679 6682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:09.513787 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "4908d193378042b4b66566b3d42c99c7"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "1a3e91d34721f90093788105c2a25ec9"
server_key_iv: "46b0872e24e8a97921b1ff513d535826"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.513864 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.526763 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:09.529114 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:09.529260 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:09.529515 6682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:09.529841 6682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:09.529897 6682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.529942 6682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:09.529961 6682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.536375 6682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39547
I20260928 08:59:09.536413 6810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39547 every 8 connection(s)
I20260928 08:59:09.536831 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:59:09.538158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6682
I20260928 08:59:09.538295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:09.538478 392 external_mini_cluster.cc:1428] Setting key 3014bbf96d0bd32ab952ab2fe88874e3
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 08:59:09.548471 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.537735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42707 (local address 127.0.98.62:36763)
0928 08:59:09.537810 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.537813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.538120 (+ 307us) server_negotiation.cc:407] Connection header received
0928 08:59:09.538960 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.538961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.538979 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.539031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.540428 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.541020 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.542585 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.542786 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.542844 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.545573 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.545574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.545575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.545582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.547096 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.547505 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.547506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.547506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.547589 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.547836 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.547836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.547837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.547949 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.548023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.548334 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.548403 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:59:09.548765 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.537721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:42707)
0928 08:59:09.537945 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.538008 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.538838 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.539108 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.539113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.539980 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.540318 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.540328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.541120 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.541121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.542490 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.542493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.542874 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.542875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.543034 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.543578 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.543591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.545457 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.547193 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.547200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.547202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.547429 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.547672 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.547673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.547673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.547776 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.548052 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.548058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.548200 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.548304 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.548445 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260928 08:59:09.549355 6811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260928 08:59:09.549522 6811 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:09.549713 6811 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260928 08:59:09.550181 6486 ts_manager.cc:194] Registered new tserver with Master: 4908d193378042b4b66566b3d42c99c7 (127.0.98.2:39547)
I20260928 08:59:09.550588 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 08:59:09.567538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:09.647759 6816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:09.648021 6816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:09.648063 6816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:09.649247 6816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:09.649323 6816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:09.649369 6816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:09.650828 6816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:09.651111 6816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:09.651350 6816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:09.656903 6821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.656903 6824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.656982 6822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.657321 6816 server_base.cc:1034] running on GCE node
I20260928 08:59:09.657526 6816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:09.657778 6816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:09.658983 6816 hybrid_clock.cc:648] HybridClock initialized: now 1790585949658951 us; error 41 us; skew 500 ppm
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:09.661749 6816 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 08:59:09.662279 6816 webserver.cc:460] Webserver started at http://127.0.98.3:43081/ using document root <none> and password file <none>
I20260928 08:59:09.662513 6816 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.662581 6816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.662731 6816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:09.663789 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7968924dbfe4876a603b11c09f3be2f"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "3dadcc5eaacbb58af5ad5e7230a2c799"
server_key_iv: "cdce74922dead776f51c4fdd7b98cc14"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.664170 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7968924dbfe4876a603b11c09f3be2f"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "3dadcc5eaacbb58af5ad5e7230a2c799"
server_key_iv: "cdce74922dead776f51c4fdd7b98cc14"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.665588 6816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:09.666498 6831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.666705 6816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:09.666817 6816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "c7968924dbfe4876a603b11c09f3be2f"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "3dadcc5eaacbb58af5ad5e7230a2c799"
server_key_iv: "cdce74922dead776f51c4fdd7b98cc14"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.666895 6816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.685398 6816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:09.687851 6816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:09.688030 6816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:09.688292 6816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:09.688671 6816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:09.688728 6816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.688773 6816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:09.688797 6816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.694554 6816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43215
I20260928 08:59:09.694667 6944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43215 every 8 connection(s)
I20260928 08:59:09.694962 6816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:09.702726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6816
I20260928 08:59:09.702870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:09.703109 392 external_mini_cluster.cc:1428] Setting key 1787e67480e19fa0df8774581a88edb3
I20260928 08:59:09.705135 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.695637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59147 (local address 127.0.98.62:36763)
0928 08:59:09.695681 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.695684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.695904 (+ 220us) server_negotiation.cc:407] Connection header received
0928 08:59:09.696451 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.696453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.696466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.696523 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.697540 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.698116 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.699315 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.699526 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.699658 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.701973 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.701974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.701975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.701983 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.703610 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.704069 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.704071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.704071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.704152 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.704424 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.704425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.704425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.704551 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.704659 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.704950 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.705040 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 08:59:09.705353 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.695616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:59147)
0928 08:59:09.695773 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.695817 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.696339 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.696605 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.696608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.697225 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.697433 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.697438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.698216 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.698217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.699198 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.699201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.699666 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.699668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.699813 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.700362 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.700371 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.701869 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.703671 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.703678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.703681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.704001 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.704226 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.704227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.704227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.704348 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.704654 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.704661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.704812 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.704927 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.705095 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 08:59:09.705816 6945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260928 08:59:09.705914 6945 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:09.706061 6945 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260928 08:59:09.706419 6486 ts_manager.cc:194] Registered new tserver with Master: c7968924dbfe4876a603b11c09f3be2f (127.0.98.3:43215)
I20260928 08:59:09.706804 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59147
I20260928 08:59:09.715292 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 08:59:09.743501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43835
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36763,127.0.98.61:43835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:09.826951 6950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:09.827231 6950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:09.827265 6950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:09.828434 6950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:09.828508 6950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:09.828533 6950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:09.828553 6950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:09.828608 6950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:09.830147 6950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36763,127.0.98.61:43835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
W20260928 08:59:09.830302 6950 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36763,127.0.98.61:43835'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 08:59:09.830462 6950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:09.830693 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:09.835510 6958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.835557 6950 server_base.cc:1034] running on GCE node
W20260928 08:59:09.835502 6955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.835525 6956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.835952 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:09.836263 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:09.837404 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790585949837380 us; error 52 us; skew 500 ppm
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:09.839959 6950 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 08:59:09.840495 6950 webserver.cc:460] Webserver started at http://127.0.98.61:45673/ using document root <none> and password file <none>
I20260928 08:59:09.840751 6950 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.840798 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.840891 6950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:09.841907 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/instance:
uuid: "5735da49e7e440ec9b55cf38854a6a30"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "dd6fb2f93cde51e13d2d10f4597f5842"
server_key_iv: "ce0a58d8cf487d6bbb0a016b845bb396"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.842217 6950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal/instance:
uuid: "5735da49e7e440ec9b55cf38854a6a30"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "dd6fb2f93cde51e13d2d10f4597f5842"
server_key_iv: "ce0a58d8cf487d6bbb0a016b845bb396"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.843467 6950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 08:59:09.844128 6965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.844336 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:09.844442 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
uuid: "5735da49e7e440ec9b55cf38854a6a30"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "dd6fb2f93cde51e13d2d10f4597f5842"
server_key_iv: "ce0a58d8cf487d6bbb0a016b845bb396"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.844521 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.879374 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:09.881871 6950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:09.882050 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:09.886283 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43835
I20260928 08:59:09.886353 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43835 every 8 connection(s)
I20260928 08:59:09.886727 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
I20260928 08:59:09.887737 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:09.888615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 6950
I20260928 08:59:09.890132 7018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 08:59:09.899878 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.890948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.0.98.62:36763)
0928 08:59:09.891001 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.891003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.891130 (+ 127us) server_negotiation.cc:407] Connection header received
0928 08:59:09.891244 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.891245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.891256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.891313 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.891904 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.892373 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.893431 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.893629 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.893710 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.896117 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.896119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.896119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.896130 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.898235 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.898735 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.898736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.898737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.898808 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.899149 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.899150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.899150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.899348 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.899425 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.899680 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.899770 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 08:59:09.900182 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.889385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.0.98.61:43835)
0928 08:59:09.889573 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.889584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.889621 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:59:09.890169 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.890178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.890245 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.890422 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.891239 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.892387 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.893238 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.893633 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.893758 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.896613 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.896620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.896620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.896626 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.898253 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.898743 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.898746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.898747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.898855 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.899102 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.899103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.899103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.899365 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.899590 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.899685 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.899813 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260928 08:59:09.900182 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.890827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52834)
0928 08:59:09.891005 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.891060 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.891150 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.891346 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.891349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.891696 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.891812 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.891819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.892476 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.892477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.893341 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.893343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.893714 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.893716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.893775 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.894390 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.894399 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.895979 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.898325 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.898382 (+ 57us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.898385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.898656 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.898901 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.898902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.898902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.899092 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.899476 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.899482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.899553 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.899661 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.899880 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":65,"thread_start_us":82,"threads_started":1}
I20260928 08:59:09.900928 7018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 08:59:09.907721 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.901232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.0.1:57664)
0928 08:59:09.901287 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.901297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.901339 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.901466 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.901466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.901608 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.901651 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.901652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.902291 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.902292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.902818 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.902820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.903201 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.903202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.903270 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.903735 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.903740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.905282 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.906795 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.906796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.906796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.906960 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.907171 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.907172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.907172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.907231 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.907450 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.907451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.907495 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.907546 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.907633 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 08:59:09.907770 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.901296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.98.61:43835)
0928 08:59:09.901327 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.901328 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.901336 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:09.901394 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.901395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.901404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.901463 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.901735 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.902212 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.902900 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.903118 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.903203 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.905392 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.905394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.905394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.905401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.906722 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.907062 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.907063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.907063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.907112 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.907286 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.907287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.907287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.907395 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.907445 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.907563 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.907675 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":16}
I20260928 08:59:09.909263 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Bootstrap starting.
I20260928 08:59:09.910130 7018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:09.910418 7018 log.cc:824] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:09.911206 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: No bootstrap required, opened a new log
I20260928 08:59:09.912964 7018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } }
I20260928 08:59:09.913092 7018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:09.913156 7018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5735da49e7e440ec9b55cf38854a6a30, State: Initialized, Role: FOLLOWER
I20260928 08:59:09.913296 7018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } }
I20260928 08:59:09.913620 7018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:09.913580 7024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } } }
I20260928 08:59:09.913699 7024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:59:09.915477 7035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:59:09.915549 7035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:59:09.917191 6950 master_runner.cc:428] Detected that this master 5735da49e7e440ec9b55cf38854a6a30 is joining an existing cluster
I20260928 08:59:09.917246 6950 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43835
I20260928 08:59:09.924995 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.918344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.0.98.62:36763)
0928 08:59:09.918392 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.918394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.918553 (+ 159us) server_negotiation.cc:407] Connection header received
0928 08:59:09.918620 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.918620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.918632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.918693 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.919160 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.919655 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.920414 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.920607 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.920754 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.921955 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.921956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.921956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.921963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.923591 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924048 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.924049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.924050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.924110 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924334 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.924335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.924335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.924483 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.924564 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.924808 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.924893 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:09.925288 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.918276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52842)
0928 08:59:09.918495 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.918506 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.918566 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.918698 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.918699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.918987 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.919062 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.919064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.919739 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.919741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.920318 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.920320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.920766 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.920767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.920836 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.921324 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.921332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.921661 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.923687 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.923688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.923689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.923961 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.924181 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.924182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.924239 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.924593 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.924594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.924681 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.925077 (+ 396us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.925176 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":2,"thread_start_us":151,"threads_started":1}
I20260928 08:59:09.925380 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.918458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.0.98.61:43835)
0928 08:59:09.918496 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.918498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.918737 (+ 239us) server_negotiation.cc:407] Connection header received
0928 08:59:09.918760 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.918761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.918770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.918832 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.919341 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.920053 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.921116 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.921392 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.921429 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.922435 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.922436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.922436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.922442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.924203 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924523 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.924525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.924525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.924571 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924837 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.924838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.924838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.924961 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.925035 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.925188 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.925265 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:59:09.925683 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.918277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.0.1:57674)
0928 08:59:09.918531 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.918542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.918617 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.918860 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.918861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.919173 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.919240 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.919242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.920149 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.920150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.921025 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.921027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.921504 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.921505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.921574 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.921955 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.921960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.922344 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.924284 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.924285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.924454 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.924710 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.924711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.924711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.924764 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.925027 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.925028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.925065 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.925304 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.925561 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 08:59:09.935847 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.927874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.0.98.61:43835)
0928 08:59:09.927920 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.927922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.928106 (+ 184us) server_negotiation.cc:407] Connection header received
0928 08:59:09.928243 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.928243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.928252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.928296 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.928887 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.929492 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.930671 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.930851 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.930926 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.933151 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.933152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.933153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.933160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.934614 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.935027 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.935028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.935028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.935086 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.935312 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.935313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.935313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.935420 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.935493 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.935672 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.935749 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 08:59:09.935863 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.927810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.0.1:57676)
0928 08:59:09.927996 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.928046 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.928129 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.928303 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.928306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.928714 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.928779 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.928781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.929585 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.929586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.930577 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.930580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.930930 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.930931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.931030 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.931491 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.931496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.933060 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.934718 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.934726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.934729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.934937 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.935180 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.935181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.935181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.935245 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.935485 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.935492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.935547 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.935649 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.935768 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260928 08:59:09.936663 6486 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43835: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43835
}
attrs {
promote: true
}
}
dest_uuid: "f2dbcf189fc949769c80262bc045854d"
cas_config_opid_index: -1
I20260928 08:59:09.937000 6486 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } }
I20260928 08:59:09.937538 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5735da49e7e440ec9b55cf38854a6a30 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } }
I20260928 08:59:09.937826 6992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:09.938624 7044 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43835
I20260928 08:59:09.938716 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } }
I20260928 08:59:09.938791 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:09.938796 6992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Refusing update from remote peer f2dbcf189fc949769c80262bc045854d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 08:59:09.939122 6950 master.cc:537] Master@127.0.98.61:43835 shutting down...
I20260928 08:59:09.939123 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:59:09.939836 6470 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 08:59:09.940449 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d -> Peer 5735da49e7e440ec9b55cf38854a6a30 (127.0.98.61:43835): Couldn't send request to peer 5735da49e7e440ec9b55cf38854a6a30. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:59:09.941838 6950 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:09.941934 6950 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:09.941970 6950 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: stopping tablet replica
I20260928 08:59:09.944548 6950 master.cc:552] Master@127.0.98.61:43835 shutdown complete.
I20260928 08:59:09.944588 6950 master_runner.cc:305] Clearing existing system tablet
I20260928 08:59:09.944777 6950 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.944818 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.945434 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:09.945866 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.945986 6950 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.946043 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.946064 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
uuid: "5735da49e7e440ec9b55cf38854a6a30"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "dd6fb2f93cde51e13d2d10f4597f5842"
server_key_iv: "ce0a58d8cf487d6bbb0a016b845bb396"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.946662 6950 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:09.948154 6950 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:59:09.948216 6950 log.cc:1196] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 08:59:09.948361 6950 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Deleting consensus metadata
I20260928 08:59:09.948482 6950 master_runner.cc:315] Copying system tablet from 127.0.98.62:36763
I20260928 08:59:09.949188 6950 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36763
I20260928 08:59:09.956239 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.949491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52854)
0928 08:59:09.949645 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:09.949661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:09.949740 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:09.949882 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:09.949882 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:09.950090 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:09.950133 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.950134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.951078 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.951079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.951765 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:09.951767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.952235 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.952236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:09.952275 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.952828 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:09.952833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:09.953187 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:09.955119 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.955120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.955121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.955328 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.955597 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:09.955598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:09.955598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:09.955661 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:09.955953 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:09.955954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:09.956010 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 08:59:09.956067 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 08:59:09.956160 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260928 08:59:09.956578 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:09.949611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.62:36763)
0928 08:59:09.949665 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:09.949667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:09.949725 (+ 58us) server_negotiation.cc:407] Connection header received
0928 08:59:09.949791 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:09.949792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:09.949803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:09.949860 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:09.950209 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.950989 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.951850 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:09.952160 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:09.952233 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:09.953269 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:09.953270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:09.953271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:09.953278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:09.955014 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.955432 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.955433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.955433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.955498 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:09.955736 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:09.955736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:09.955737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:09.955878 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:09.955946 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:09.956112 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 08:59:09.956191 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:59:09.957067 6506 tablet_copy_service.cc:140] P f2dbcf189fc949769c80262bc045854d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5735da49e7e440ec9b55cf38854a6a30 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52854)
I20260928 08:59:09.957185 6506 tablet_copy_service.cc:161] P f2dbcf189fc949769c80262bc045854d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5735da49e7e440ec9b55cf38854a6a30 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52854: session id = 5735da49e7e440ec9b55cf38854a6a30-00000000000000000000000000000000
I20260928 08:59:09.958117 6506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:59:09.958796 6950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:59:09.959909 6950 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: tablet copy: Starting download of 0 data blocks...
I20260928 08:59:09.960217 6950 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: tablet copy: Starting download of 1 WAL segments...
I20260928 08:59:09.961208 6950 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:59:09.962038 6506 tablet_copy_service.cc:342] P f2dbcf189fc949769c80262bc045854d: Request end of tablet copy session 5735da49e7e440ec9b55cf38854a6a30-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52854
I20260928 08:59:09.962101 6506 tablet_copy_service.cc:433] P f2dbcf189fc949769c80262bc045854d: ending tablet copy session 5735da49e7e440ec9b55cf38854a6a30-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5735da49e7e440ec9b55cf38854a6a30
I20260928 08:59:09.964509 6950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:09.965741 7057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.965794 7056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:09.965965 7059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:09.965983 6950 server_base.cc:1034] running on GCE node
I20260928 08:59:09.966180 6950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:09.966311 6950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:09.967442 6950 hybrid_clock.cc:648] HybridClock initialized: now 1790585949967431 us; error 40 us; skew 500 ppm
Sep 28 08:59:09 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:09.969031 6950 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 08:59:09.969384 6950 webserver.cc:460] Webserver started at http://127.0.98.61:46511/ using document root <none> and password file <none>
I20260928 08:59:09.969540 6950 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:09.969587 6950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:09.970398 6950 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:09.971024 7066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:09.971167 6950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:09.971217 6950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
uuid: "5735da49e7e440ec9b55cf38854a6a30"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "dd6fb2f93cde51e13d2d10f4597f5842"
server_key_iv: "ce0a58d8cf487d6bbb0a016b845bb396"
server_key_version: "encryptionkey@0"
I20260928 08:59:09.971287 6950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:09.981781 6950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:09.983752 6950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:09.987198 6950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43835
I20260928 08:59:09.987528 7118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43835 every 8 connection(s)
I20260928 08:59:09.987656 6950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
I20260928 08:59:09.988565 7119 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:59:09.989130 7119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Bootstrap starting.
I20260928 08:59:09.992019 7119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:59:09.992259 7119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: Bootstrap complete.
I20260928 08:59:09.992473 7119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } }
I20260928 08:59:09.992600 7119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5735da49e7e440ec9b55cf38854a6a30, State: Initialized, Role: LEARNER
I20260928 08:59:09.992686 7119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } }
I20260928 08:59:09.992878 7119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:09.992885 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } }
I20260928 08:59:09.993036 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:59:09.994442 7133 catalog_manager.cc:1259] Loaded cluster ID: 3410b440408d4eddb5e3bbad63cce68f
I20260928 08:59:09.994491 7133 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: loading cluster ID for follower catalog manager: success
I20260928 08:59:09.995499 7133 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: acquiring CA information for follower catalog manager: success
I20260928 08:59:09.995787 7133 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:59:10.009732 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.002118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.0.98.61:43835)
0928 08:59:10.002224 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.002225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.002234 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:10.002256 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.002257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.002266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.002335 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.003084 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.003759 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.004846 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.005069 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.005152 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.006606 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.006607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.006608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.006614 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.008347 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.008849 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.008850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.008851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.008913 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.009210 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.009211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.009211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.009323 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.009403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.009569 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.009643 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 08:59:10.037088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46771
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:10.122586 7137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:10.122814 7137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:10.122838 7137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:10.123956 7137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:10.123997 7137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:10.124013 7137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:10.124027 7137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:10.124042 7137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:10.125499 7137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:10.125698 7137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:10.125880 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:10.130793 7143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.130910 7137 server_base.cc:1034] running on GCE node
W20260928 08:59:10.130792 7145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.130731 7142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.131225 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:10.131523 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:10.132712 7137 hybrid_clock.cc:648] HybridClock initialized: now 1790585950132699 us; error 29 us; skew 500 ppm
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:10.135313 7137 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 08:59:10.135694 7137 webserver.cc:460] Webserver started at http://127.0.98.60:39565/ using document root <none> and password file <none>
I20260928 08:59:10.135866 7137 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:10.135905 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:10.135994 7137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:10.137092 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/instance:
uuid: "4e21e8e13c084226b372376a184c8dd7"
format_stamp: "Formatted at 2026-09-28 08:59:10 on dist-test-slave-286n"
server_key: "93a2396dc282147a64bcd66ed7aa8d04"
server_key_iv: "9988feb486d633ef9120fe224b6f5a85"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.137384 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal/instance:
uuid: "4e21e8e13c084226b372376a184c8dd7"
format_stamp: "Formatted at 2026-09-28 08:59:10 on dist-test-slave-286n"
server_key: "93a2396dc282147a64bcd66ed7aa8d04"
server_key_iv: "9988feb486d633ef9120fe224b6f5a85"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.138660 7137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:59:10.139401 7152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.139569 7137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:10.139660 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
uuid: "4e21e8e13c084226b372376a184c8dd7"
format_stamp: "Formatted at 2026-09-28 08:59:10 on dist-test-slave-286n"
server_key: "93a2396dc282147a64bcd66ed7aa8d04"
server_key_iv: "9988feb486d633ef9120fe224b6f5a85"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.139715 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:10.155611 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:10.157938 7137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:10.158085 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:10.162776 7137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46771
I20260928 08:59:10.162863 7204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46771 every 8 connection(s)
I20260928 08:59:10.163214 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
I20260928 08:59:10.164177 7205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:10.165769 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:10.172348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 7137
I20260928 08:59:10.175300 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.166630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.0.98.62:36763)
0928 08:59:10.166699 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.166701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.166798 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:59:10.167220 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.167221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.167229 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.167289 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.167849 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.168341 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.169521 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.169715 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.169768 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.172158 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.172160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.172160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.172167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.173884 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.174318 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.174319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.174319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.174384 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.174648 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.174649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.174649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.174771 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.174850 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.175121 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.175208 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:59:10.175499 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.166509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52856)
0928 08:59:10.166663 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.166706 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.167140 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.167340 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.167343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.167586 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.167780 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.167784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.168435 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.168436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.169405 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.169407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.169773 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.169774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.169874 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.170458 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.170467 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.172031 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.173990 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.173997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.173999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.174229 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.174458 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.174459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.174459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.174577 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.174841 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.174847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.174989 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.175099 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.175256 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 08:59:10.176205 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 08:59:10.181782 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.172882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.98.60:46771)
0928 08:59:10.173065 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.173070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.173096 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:59:10.173114 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.173115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.173127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.173204 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.173792 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.174487 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.175258 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.175731 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.175807 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.178751 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.178756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.178757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.178762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.180383 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.180838 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.180841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.180841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.180911 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.181122 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.181123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.181124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.181302 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.181418 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.181569 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.181651 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260928 08:59:10.183239 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.176473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.0.1:57690)
0928 08:59:10.176528 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.176540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.176582 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.176791 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.176791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.176899 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.176940 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.176941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.177850 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.177851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.178540 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.178542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.178963 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.178964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.179037 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.179335 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.179340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.180622 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.182429 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.182429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.182430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.182601 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.182768 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.182769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.182769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.182814 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.183007 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.183007 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.183068 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.183119 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.183183 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 08:59:10.183305 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.176561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.0.98.61:43835)
0928 08:59:10.176655 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.176657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.176667 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:10.176706 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.176707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.176722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.176781 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.177014 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.177754 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.178628 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.178893 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.178941 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.180737 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.180738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.180739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.180746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.182355 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.182664 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.182664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.182665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.182712 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.182865 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.182865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.182866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.182957 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.183004 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.183152 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.183216 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:59:10.183622 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46771 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 08:59:10.191625 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.183873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.0.1:39320)
0928 08:59:10.183920 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.183930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.183967 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.184079 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.184080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.184206 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.184230 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.184231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.184906 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.184907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.185374 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.185375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.185750 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.185751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.185796 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.186203 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.186208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.187375 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.188772 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.188773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.188773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.188927 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.189174 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.189175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.189175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.189227 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.191303 (+ 2076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.191304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.191340 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.191400 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.191493 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 08:59:10.191629 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.183944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.98.60:46771)
0928 08:59:10.183977 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.183979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.183987 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:10.184006 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.184006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.184016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.184075 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.184300 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.184826 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.185467 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.185671 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.185752 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.187472 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.187473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.187473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.187480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.188701 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.189009 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.189010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.189010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.189098 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.191124 (+ 2026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.191125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.191125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.191220 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.191313 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.191430 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.191541 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":19}
I20260928 08:59:10.193506 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Bootstrap starting.
I20260928 08:59:10.194301 7205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:10.194628 7205 log.cc:824] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:10.197302 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: No bootstrap required, opened a new log
I20260928 08:59:10.198982 7205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46771 } }
I20260928 08:59:10.199251 7205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:10.199294 7205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21e8e13c084226b372376a184c8dd7, State: Initialized, Role: FOLLOWER
I20260928 08:59:10.199425 7205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46771 } }
I20260928 08:59:10.199734 7205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:10.200811 7210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46771 } } }
I20260928 08:59:10.200914 7210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:59:10.202633 7221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:59:10.202693 7221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:59:10.206152 7137 master_runner.cc:428] Detected that this master 4e21e8e13c084226b372376a184c8dd7 is joining an existing cluster
I20260928 08:59:10.206192 7137 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46771
I20260928 08:59:10.214998 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.207612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.0.98.62:36763)
0928 08:59:10.207652 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.207654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.207899 (+ 245us) server_negotiation.cc:407] Connection header received
0928 08:59:10.207998 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.207999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.208009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.208040 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.208525 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.209286 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.210342 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.210991 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.211036 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.212544 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.212545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.212545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.212552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.213952 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.214334 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.214335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.214335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.214384 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.214582 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.214582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.214583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.214680 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.214730 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.214880 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.214935 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:59:10.215320 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.207693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52866)
0928 08:59:10.207834 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.207845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.207925 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.208113 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.208114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.208339 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.208414 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.208416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.209382 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.209383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.210242 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.210244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.211114 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.211115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.211176 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.212015 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.212020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.212444 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.214031 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.214033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.214033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.214259 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.214447 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.214448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.214448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.214521 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.214741 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.214742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.214788 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.214866 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.214966 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 08:59:10.233182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6548
I20260928 08:59:10.237567 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.207354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.0.1:39330)
0928 08:59:10.207818 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.207829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.207898 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.208038 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.208038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.208249 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.208303 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.208305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.209137 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.209138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.209910 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.209911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.210350 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.210350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.210405 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.233409 (+ 23004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.233415 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.234099 (+ 684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.236212 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.236213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.236213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.236421 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.236593 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.236593 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.236593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.236663 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.237221 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.237222 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.237273 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.237323 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.237429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":60,"threads_started":1}
I20260928 08:59:10.237968 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.207503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.0.98.60:46771)
0928 08:59:10.207542 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.207544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.207877 (+ 333us) server_negotiation.cc:407] Connection header received
0928 08:59:10.207961 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.207961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.207969 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.207994 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.208378 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.209063 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.209977 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.210252 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.210286 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.234186 (+ 23900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.234187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.234187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.234194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.236117 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.236489 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.236490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.236491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.236533 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.236744 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.236745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.236745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.236897 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.237765 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.237804 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.237876 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":22}
I20260928 08:59:10.239015 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.207220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.98.61:43835)
0928 08:59:10.207261 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.207263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.207350 (+ 87us) server_negotiation.cc:407] Connection header received
0928 08:59:10.207412 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.207413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.207424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.207499 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.207948 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.208775 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.209832 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.210121 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.210158 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.233582 (+ 23424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.233584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.233584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.233592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.236342 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.237008 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.237009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.237010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.237068 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.237789 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.237790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.237790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.237928 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.237969 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.238790 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.238875 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:59:10.239377 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.207154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.0.1:57700)
0928 08:59:10.207278 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.207289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.207347 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.207554 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.207555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.207770 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.207855 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.207857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.208862 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.208863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.209744 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.209745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.210240 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.210241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.210297 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.210781 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.210785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.233437 (+ 22652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.236453 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.236454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.236454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.236709 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.237161 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.237162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.237162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.237233 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.238127 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.238128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.238169 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.239102 (+ 933us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.239218 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260928 08:59:10.240844 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.217435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.0.98.60:46771)
0928 08:59:10.217601 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.217603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.217611 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:10.217650 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.217651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.217659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.217713 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.218004 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.218487 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.219438 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.219623 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.219672 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.221554 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.221556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.221556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.221563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.238672 (+ 17109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.239852 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.239853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.239853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.239913 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.240196 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.240197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.240198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.240342 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.240457 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.240665 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.240747 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":127,"threads_started":1}
I20260928 08:59:10.240839 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.217377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.0.1:39346)
0928 08:59:10.217424 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.217433 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.217465 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.217734 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.217734 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.217895 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.217935 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.217937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.218559 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.218560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.219359 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.219361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.219690 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.219691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.219756 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.220106 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.220111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.221465 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.239500 (+ 18035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.239502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.239502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.239766 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.239998 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.239999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.240000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.240091 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.240465 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.240466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.240523 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.240607 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.240747 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 08:59:10.240906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35911
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36763
--webserver_port=38355
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:59:10.241297 6486 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46771: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4e21e8e13c084226b372376a184c8dd7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46771
}
attrs {
promote: true
}
}
dest_uuid: "f2dbcf189fc949769c80262bc045854d"
cas_config_opid_index: 5
I20260928 08:59:10.241528 6486 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:10.241972 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4e21e8e13c084226b372376a184c8dd7 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.242269 7179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:10.243017 7045 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46771
I20260928 08:59:10.243094 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.243153 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:10.243157 7179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 1 FOLLOWER]: Refusing update from remote peer f2dbcf189fc949769c80262bc045854d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:59:10.243501 7137 master.cc:537] Master@127.0.98.60:46771 shutting down...
I20260928 08:59:10.243835 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:59:10.245147 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d -> Peer 4e21e8e13c084226b372376a184c8dd7 (127.0.98.60:46771): Couldn't send request to peer 4e21e8e13c084226b372376a184c8dd7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:59:10.247515 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.242366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.0.98.61:43835)
0928 08:59:10.242407 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.242408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.243642 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 08:59:10.243665 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.243665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.243687 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.243741 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:10.244232 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.245006 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.246062 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.246691 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.246761 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.246840 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.247393 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.247430 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:59:10.247915 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.241965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.0.1:57704)
0928 08:59:10.243525 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.243537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.243587 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.244095 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.244097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.244098 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:10.244148 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.244149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.245115 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.245116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.245945 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.245947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.246873 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.246874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.247261 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.247291 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.247346 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.247834 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543}
I20260928 08:59:10.248050 7137 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:10.248138 7137 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:10.248188 7137 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: stopping tablet replica
I20260928 08:59:10.248261 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 LEARNER]: Refusing update from remote peer f2dbcf189fc949769c80262bc045854d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:59:10.248466 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 08:59:10.249032 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4e21e8e13c084226b372376a184c8dd7 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.249974 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.251025 7044 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: attempting to promote NON_VOTER 5735da49e7e440ec9b55cf38854a6a30 to VOTER
I20260928 08:59:10.251240 7044 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:10.250357 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2dbcf189fc949769c80262bc045854d. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: true } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.251550 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:59:10.251667 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:59:10.252123 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.252013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.0.1:39362)
0928 08:59:10.252056 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.252096 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 08:59:10.252352 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 LEARNER]: Refusing update from remote peer f2dbcf189fc949769c80262bc045854d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 08:59:10.252592 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.252331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.0.1:39370)
0928 08:59:10.252525 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.252552 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 08:59:10.252532 7234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 08:59:10.252774 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d -> Peer 4e21e8e13c084226b372376a184c8dd7 (127.0.98.60:46771): Couldn't send request to peer 4e21e8e13c084226b372376a184c8dd7. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 08:59:10.253280 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5735da49e7e440ec9b55cf38854a6a30 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.254041 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.254120 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:10.254175 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.254222 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:10.254446 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5735da49e7e440ec9b55cf38854a6a30 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.255425 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2dbcf189fc949769c80262bc045854d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.255530 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:59:10.262914 7137 master.cc:552] Master@127.0.98.60:46771 shutdown complete.
I20260928 08:59:10.262981 7137 master_runner.cc:305] Clearing existing system tablet
I20260928 08:59:10.263121 7137 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:10.263193 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:10.263814 7137 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:10.264278 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.264451 7137 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:10.264504 7137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.264526 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
uuid: "4e21e8e13c084226b372376a184c8dd7"
format_stamp: "Formatted at 2026-09-28 08:59:10 on dist-test-slave-286n"
server_key: "93a2396dc282147a64bcd66ed7aa8d04"
server_key_iv: "9988feb486d633ef9120fe224b6f5a85"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.265307 7137 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:10.266594 7137 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:59:10.266674 7137 log.cc:1196] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 08:59:10.266865 7137 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Deleting consensus metadata
I20260928 08:59:10.266997 7137 master_runner.cc:315] Copying system tablet from 127.0.98.62:36763
I20260928 08:59:10.267637 7137 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36763
I20260928 08:59:10.274320 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.267996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.0.98.62:36763)
0928 08:59:10.268045 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.268047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.268166 (+ 119us) server_negotiation.cc:407] Connection header received
0928 08:59:10.268247 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.268248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.268258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.268303 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.268614 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.269147 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.270085 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.270286 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.270382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.271597 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.271598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.271598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.271606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.273174 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.273536 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.273537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.273537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.273598 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.273808 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.273809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.273810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.273924 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.273997 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.274151 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.274228 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:59:10.274335 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.267955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52882)
0928 08:59:10.268087 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.268097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.268182 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.268316 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.268316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.268486 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.268527 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.268529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.269231 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.269232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.269995 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.269997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.270384 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.270385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.270468 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.271015 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.271021 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.271472 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.273276 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.273277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.273278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.273453 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.273664 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.273665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.273665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.273727 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.273989 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.273990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.274034 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.274123 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.274228 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260928 08:59:10.274859 6506 tablet_copy_service.cc:140] P f2dbcf189fc949769c80262bc045854d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4e21e8e13c084226b372376a184c8dd7 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52882)
I20260928 08:59:10.274935 6506 tablet_copy_service.cc:161] P f2dbcf189fc949769c80262bc045854d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4e21e8e13c084226b372376a184c8dd7 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52882: session id = 4e21e8e13c084226b372376a184c8dd7-00000000000000000000000000000000
I20260928 08:59:10.275460 6506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:59:10.275899 7137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:59:10.276942 7137 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: tablet copy: Starting download of 0 data blocks...
I20260928 08:59:10.277108 7137 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: tablet copy: Starting download of 1 WAL segments...
I20260928 08:59:10.277801 7137 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:59:10.278498 6506 tablet_copy_service.cc:342] P f2dbcf189fc949769c80262bc045854d: Request end of tablet copy session 4e21e8e13c084226b372376a184c8dd7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52882
I20260928 08:59:10.278553 6506 tablet_copy_service.cc:433] P f2dbcf189fc949769c80262bc045854d: ending tablet copy session 4e21e8e13c084226b372376a184c8dd7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4e21e8e13c084226b372376a184c8dd7
I20260928 08:59:10.280565 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:10.281960 7243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.282017 7246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.282202 7137 server_base.cc:1034] running on GCE node
W20260928 08:59:10.285174 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.285408 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:10.285578 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:10.286712 7137 hybrid_clock.cc:648] HybridClock initialized: now 1790585950286705 us; error 26 us; skew 500 ppm
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:10.288182 7137 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 08:59:10.288439 7137 webserver.cc:460] Webserver started at http://127.0.98.60:41745/ using document root <none> and password file <none>
I20260928 08:59:10.288558 7137 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:10.288594 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:10.289211 7137 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.289737 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.289878 7137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:10.289922 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
uuid: "4e21e8e13c084226b372376a184c8dd7"
format_stamp: "Formatted at 2026-09-28 08:59:10 on dist-test-slave-286n"
server_key: "93a2396dc282147a64bcd66ed7aa8d04"
server_key_iv: "9988feb486d633ef9120fe224b6f5a85"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.289988 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:10.300158 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:10.301949 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:10.305027 7137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46771
I20260928 08:59:10.305462 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
I20260928 08:59:10.306299 7306 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:59:10.306802 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Bootstrap starting.
I20260928 08:59:10.307862 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46771 every 8 connection(s)
I20260928 08:59:10.309172 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:59:10.309399 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: Bootstrap complete.
I20260928 08:59:10.309608 7306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:10.309697 7306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e21e8e13c084226b372376a184c8dd7, State: Initialized, Role: LEARNER
I20260928 08:59:10.309741 7306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:10.309913 7306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:10.309906 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } } }
I20260928 08:59:10.310106 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:59:10.311712 7320 catalog_manager.cc:1259] Loaded cluster ID: 3410b440408d4eddb5e3bbad63cce68f
I20260928 08:59:10.312245 7320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: loading cluster ID for follower catalog manager: success
I20260928 08:59:10.313102 7320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: acquiring CA information for follower catalog manager: success
I20260928 08:59:10.313372 7320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e21e8e13c084226b372376a184c8dd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:59:10.360733 7231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:10.361040 7231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:10.361083 7231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:10.362903 7231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:10.362974 7231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:10.363021 7231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:10.365410 7231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38355
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:10.365713 7231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:10.366001 7231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:10.371754 7326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.371757 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.371811 7325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.372164 7231 server_base.cc:1034] running on GCE node
I20260928 08:59:10.372357 7231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:10.372608 7231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:10.373801 7231 hybrid_clock.cc:648] HybridClock initialized: now 1790585950373788 us; error 32 us; skew 500 ppm
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:10.376426 7231 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 08:59:10.376974 7231 webserver.cc:460] Webserver started at http://127.0.98.1:38355/ using document root <none> and password file <none>
I20260928 08:59:10.377300 7231 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:10.377393 7231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:10.379005 7231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:59:10.379756 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.379961 7231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:10.380060 7231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "08db1b76e35e496a98da7649d4a35f0d"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "f2fca13932db71b272aa473410df5ee7"
server_key_iv: "4190cbfbfb3f225fc8c7ffe4f99dd4b9"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.380492 7231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:10.395546 7231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:10.397941 7231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:10.398137 7231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:10.398455 7231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:10.398854 7231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:10.398907 7231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.398950 7231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:10.398976 7231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.405849 7231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35911
I20260928 08:59:10.405936 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35911 every 8 connection(s)
I20260928 08:59:10.406286 7231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:10.407178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 7231
I20260928 08:59:10.407337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6682
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 08:59:10.418190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39547
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36763
--webserver_port=39839
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:10.419698 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.407119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.98.1:34545)
0928 08:59:10.407314 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.407367 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.408086 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.408446 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.408451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.409418 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.409719 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.409726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.410687 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.410688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.412190 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.412193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.412618 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.412619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.412790 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.413566 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.413576 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.415715 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.417948 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.417955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.417957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.418232 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.418464 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.418465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.418465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.418620 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.418920 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.418927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.419092 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.419216 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.419401 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260928 08:59:10.420186 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.407102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34545 (local address 127.0.98.60:46771)
0928 08:59:10.407271 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.407273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.408225 (+ 952us) server_negotiation.cc:407] Connection header received
0928 08:59:10.408256 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.408257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.408271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.408330 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.409811 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.410558 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.412312 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.412526 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.412592 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.415826 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.415828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.415829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.415837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.417879 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.418331 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.418333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.418333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.418396 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.418696 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.418697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.418697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.418832 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.418892 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.419967 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.420071 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 08:59:10.421131 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46771
I20260928 08:59:10.421262 7449 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.421464 7449 heartbeater.cc:511] Master 127.0.98.60:46771 requested a full tablet report, sending...
I20260928 08:59:10.422003 7270 ts_manager.cc:194] Registered new tserver with Master: 08db1b76e35e496a98da7649d4a35f0d (127.0.98.1:35911)
I20260928 08:59:10.422859 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.407612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:46653)
0928 08:59:10.413394 (+ 5782us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.413403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.413458 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.413675 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.413675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.413956 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.414033 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.414034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.414956 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.414957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.416481 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.416493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.417047 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.417049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.417114 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.417479 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.417493 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.419180 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.421358 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.421358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.421359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.421890 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.422148 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.422148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.422149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.422212 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.422471 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.422472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.422520 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.422617 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.422724 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5758,"mutex_wait_us":663,"thread_start_us":61,"threads_started":1}
I20260928 08:59:10.423055 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.407946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46653 (local address 127.0.98.62:36763)
0928 08:59:10.408130 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.408132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.413460 (+ 5328us) server_negotiation.cc:407] Connection header received
0928 08:59:10.413557 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.413559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.413573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.413625 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.414125 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.414857 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.416582 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.416922 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.416967 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.419264 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.419266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.419266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.419273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.421295 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.421981 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.421983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.421983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.422042 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.422286 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.422287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.422287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.422403 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.422480 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.422874 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.422963 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260928 08:59:10.423565 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260928 08:59:10.423642 7450 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.423717 7450 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260928 08:59:10.423969 6486 ts_manager.cc:194] Re-registered known tserver with Master: 08db1b76e35e496a98da7649d4a35f0d (127.0.98.1:35911)
I20260928 08:59:10.424355 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46653
I20260928 08:59:10.424904 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.407931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60905 (local address 127.0.98.61:43835)
0928 08:59:10.407979 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.407982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.413321 (+ 5339us) server_negotiation.cc:407] Connection header received
0928 08:59:10.413368 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.413370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.413383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.413439 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.414069 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.414896 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.415804 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.416111 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.416477 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.421280 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.421282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.421282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.421291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.423572 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.424028 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.424029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.424030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.424089 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.424301 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.424302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.424302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.424458 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.424504 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.424733 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.424817 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:59:10.424968 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.407430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.98.1:60905)
0928 08:59:10.413227 (+ 5797us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.413239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.413308 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.413578 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.413579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.413893 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.413960 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.413962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.414980 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.414980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.415702 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.415704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.416533 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.416534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.416604 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.417486 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.417490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.421166 (+ 3676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.423672 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.423673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.423673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.423921 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.424167 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.424168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.424168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.424227 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.424556 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.424557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.424602 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.424701 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.424817 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5775,"mutex_wait_us":40,"thread_start_us":59,"threads_started":1}
I20260928 08:59:10.425324 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43835
I20260928 08:59:10.425376 7451 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.425439 7451 heartbeater.cc:511] Master 127.0.98.61:43835 requested a full tablet report, sending...
I20260928 08:59:10.425753 7083 ts_manager.cc:194] Registered new tserver with Master: 08db1b76e35e496a98da7649d4a35f0d (127.0.98.1:35911)
W20260928 08:59:10.503233 7457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:10.503496 7457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:10.503530 7457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:10.504968 7457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:10.505014 7457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:10.505043 7457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:10.506497 7457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39839
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:10.506690 7457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:10.506872 7457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:10.512053 7463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.512097 7462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.512182 7465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.512372 7457 server_base.cc:1034] running on GCE node
I20260928 08:59:10.512583 7457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:10.512846 7457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:10.514005 7457 hybrid_clock.cc:648] HybridClock initialized: now 1790585950513992 us; error 44 us; skew 500 ppm
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:10.516475 7457 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 08:59:10.516970 7457 webserver.cc:460] Webserver started at http://127.0.98.2:39839/ using document root <none> and password file <none>
I20260928 08:59:10.517172 7457 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:10.517221 7457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:10.518543 7457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:10.519114 7472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.519335 7457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:10.519410 7457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "4908d193378042b4b66566b3d42c99c7"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "1a3e91d34721f90093788105c2a25ec9"
server_key_iv: "46b0872e24e8a97921b1ff513d535826"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.519742 7457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:10.540797 7457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:10.542603 7457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:10.542711 7457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:10.542881 7457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:10.543220 7457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:10.543256 7457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.543303 7457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:10.543330 7457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.549288 7457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39547
I20260928 08:59:10.549335 7585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39547 every 8 connection(s)
I20260928 08:59:10.549669 7457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:59:10.554664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 7457
I20260928 08:59:10.554832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6816
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 08:59:10.561539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43215
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36763
--webserver_port=43081
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:59:10.562748 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.550422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40731 (local address 127.0.98.62:36763)
0928 08:59:10.550479 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.550481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.550815 (+ 334us) server_negotiation.cc:407] Connection header received
0928 08:59:10.551878 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.551879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.551892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.551925 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.552775 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.553303 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.554654 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.554971 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.555036 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.558026 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.558027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.558028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.558036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.559873 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.560360 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.560362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.560363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.560430 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.561438 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.561440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.561440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.561608 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.562142 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.562551 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.562643 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 08:59:10.563031 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.550487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45957 (local address 127.0.98.61:43835)
0928 08:59:10.550526 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.550528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.550815 (+ 287us) server_negotiation.cc:407] Connection header received
0928 08:59:10.551877 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.551879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.551892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.551925 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.553393 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.553845 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.554845 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.555119 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.555156 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.559864 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.559866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.559866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.559876 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.561846 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.562232 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.562233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.562233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.562288 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.562480 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.562481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.562482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.562617 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.562660 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.562860 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.562952 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 08:59:10.563695 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.550399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:40731)
0928 08:59:10.550684 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.550731 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.551751 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.552007 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.552012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.552334 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.552672 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.552679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.553424 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.553425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.554554 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.554556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.555113 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.555114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.555264 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.555875 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.555885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.557911 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.559964 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.559970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.559973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.560277 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.560705 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.560707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.560707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.561335 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.562198 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.562205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.562387 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.563241 (+ 854us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.563454 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":146,"thread_start_us":198,"threads_started":1}
I20260928 08:59:10.563697 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.550401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.98.2:45957)
0928 08:59:10.550731 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.550740 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.551751 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.552007 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.552012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.552406 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.552672 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.552679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.553931 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.553932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.554757 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.554760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.555401 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.555402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.555479 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.557451 (+ 1972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.557456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.559749 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.561941 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.561942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.561943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.562147 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.562356 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.562356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.562357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.562409 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.562708 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.562709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.562747 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.563169 (+ 422us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.563406 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":35,"thread_start_us":199,"threads_started":1}
I20260928 08:59:10.563896 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.550808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.98.2:48697)
0928 08:59:10.550929 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.550940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.551769 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.552005 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.552017 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.552370 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.552674 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.552680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.556467 (+ 3787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.556468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.557327 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.557331 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.558494 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.558495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.558582 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.558968 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.558973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.560766 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.562678 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.562679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.562680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.562973 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.563188 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.563189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.563189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.563250 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.563526 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.563527 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.563571 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.563642 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.563756 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":7,"thread_start_us":57,"threads_started":1}
I20260928 08:59:10.564199 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.550891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48697 (local address 127.0.98.60:46771)
0928 08:59:10.550932 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.550934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.550980 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:59:10.551857 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.551859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.551874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.551921 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.553931 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.554382 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.557977 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.558323 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.558383 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.560873 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.560875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.560875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.560884 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.562585 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.563043 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.563045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.563045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.563111 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.563314 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.563315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.563315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.563445 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.563502 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.564010 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.564110 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:10.564306 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43835
I20260928 08:59:10.564402 7588 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.564607 7588 heartbeater.cc:511] Master 127.0.98.61:43835 requested a full tablet report, sending...
I20260928 08:59:10.564702 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260928 08:59:10.564749 7587 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.564817 7587 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260928 08:59:10.564857 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46771
I20260928 08:59:10.564899 7586 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.564942 7586 heartbeater.cc:511] Master 127.0.98.60:46771 requested a full tablet report, sending...
I20260928 08:59:10.565065 6486 ts_manager.cc:194] Re-registered known tserver with Master: 4908d193378042b4b66566b3d42c99c7 (127.0.98.2:39547)
I20260928 08:59:10.565586 6486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40731
I20260928 08:59:10.566053 7083 ts_manager.cc:194] Registered new tserver with Master: 4908d193378042b4b66566b3d42c99c7 (127.0.98.2:39547)
I20260928 08:59:10.566327 7270 ts_manager.cc:194] Registered new tserver with Master: 4908d193378042b4b66566b3d42c99c7 (127.0.98.2:39547)
W20260928 08:59:10.645700 7593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:10.645905 7593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:10.645926 7593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:10.646991 7593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:10.647033 7593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:10.647059 7593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:10.648509 7593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43081
--tserver_master_addrs=127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:10.648794 7593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:10.649058 7593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:10.653990 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.653990 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:10.654001 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:10.654337 7593 server_base.cc:1034] running on GCE node
I20260928 08:59:10.654559 7593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:10.654795 7593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:10.655962 7593 hybrid_clock.cc:648] HybridClock initialized: now 1790585950655954 us; error 35 us; skew 500 ppm
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:10.658648 7593 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 08:59:10.659129 7593 webserver.cc:460] Webserver started at http://127.0.98.3:43081/ using document root <none> and password file <none>
I20260928 08:59:10.659366 7593 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:10.659449 7593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:10.660885 7593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:10.661550 7608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.661736 7593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 08:59:10.661849 7593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "c7968924dbfe4876a603b11c09f3be2f"
format_stamp: "Formatted at 2026-09-28 08:59:09 on dist-test-slave-286n"
server_key: "3dadcc5eaacbb58af5ad5e7230a2c799"
server_key_iv: "cdce74922dead776f51c4fdd7b98cc14"
server_key_version: "encryptionkey@0"
I20260928 08:59:10.662241 7593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:10.676123 7593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:10.678247 7593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:10.678411 7593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:10.678648 7593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:10.678951 7593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:10.679003 7593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.679054 7593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:10.679082 7593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:10.685127 7593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43215
I20260928 08:59:10.685174 7721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43215 every 8 connection(s)
I20260928 08:59:10.685570 7593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 08:59:10.687846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 7593
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 08:59:10 dist-test-slave-286n krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 08:59:10.699888 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.686400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39171 (local address 127.0.98.60:46771)
0928 08:59:10.686467 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.686469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.686609 (+ 140us) server_negotiation.cc:407] Connection header received
0928 08:59:10.687300 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.687301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.687315 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.687373 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.688426 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.689174 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.690792 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691100 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.691136 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.696226 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.696227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.696228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.696236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.697981 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.698638 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.698639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.698640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.698705 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.699090 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.699091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.699091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.699227 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.699275 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.699698 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.699777 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:59:10.699903 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.686781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55401 (local address 127.0.98.61:43835)
0928 08:59:10.686828 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.686830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.686841 (+ 11us) server_negotiation.cc:407] Connection header received
0928 08:59:10.687675 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.687676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.687689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.687768 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.688381 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.689186 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.690822 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691141 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.691189 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.696273 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.696274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.696275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.696283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.698181 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.698650 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.698652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.698652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.698723 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.699083 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.699084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.699085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.699241 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.699287 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.699711 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.699801 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:59:10.700546 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.686327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46771 (local address 127.0.98.3:39171)
0928 08:59:10.686495 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.686534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.687163 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.687430 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.687433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.688200 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.688294 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.688296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.689362 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.689363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.690693 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.690695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691223 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.691224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.691364 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.694260 (+ 2896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.694266 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.696123 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.698085 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.698094 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.698096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.698546 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.698785 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.698786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.698786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.698981 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.699384 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.699385 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.699534 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.700008 (+ 474us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.700263 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":92,"thread_start_us":117,"threads_started":1}
I20260928 08:59:10.700544 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.686638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43835 (local address 127.0.98.3:55401)
0928 08:59:10.686775 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.686784 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.687173 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.687822 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.687823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.688166 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.688255 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.688262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.689332 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.689333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.690697 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.690699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691297 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.691298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.691378 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.691961 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.691973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.695128 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.698252 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.698253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.698253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.698547 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.698801 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.698802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.698802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.698982 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.699356 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.699362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.699535 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.700055 (+ 520us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.700330 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":242,"thread_start_us":75,"threads_started":1}
I20260928 08:59:10.701061 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46771
I20260928 08:59:10.701193 7722 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.701424 7722 heartbeater.cc:511] Master 127.0.98.60:46771 requested a full tablet report, sending...
I20260928 08:59:10.701690 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.688946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.0.98.62:36763)
0928 08:59:10.689674 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.689676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.689876 (+ 200us) server_negotiation.cc:407] Connection header received
0928 08:59:10.689900 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.689900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.689912 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.689979 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.690803 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691504 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.693399 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.693720 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.693807 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.695291 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.695292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.695293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.695299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.699494 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.700603 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.700604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.700604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.700683 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.700904 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.700905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.700905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.701038 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.701135 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.701476 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.701562 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":80,"threads_started":1}
I20260928 08:59:10.701910 7270 ts_manager.cc:194] Registered new tserver with Master: c7968924dbfe4876a603b11c09f3be2f (127.0.98.3:43215)
I20260928 08:59:10.702704 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43835
I20260928 08:59:10.702778 7724 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.702862 7724 heartbeater.cc:511] Master 127.0.98.61:43835 requested a full tablet report, sending...
I20260928 08:59:10.703289 7083 ts_manager.cc:194] Registered new tserver with Master: c7968924dbfe4876a603b11c09f3be2f (127.0.98.3:43215)
I20260928 08:59:10.703599 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.686331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52565 (local address 127.0.98.62:36763)
0928 08:59:10.686401 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.686403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.686618 (+ 215us) server_negotiation.cc:407] Connection header received
0928 08:59:10.687430 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.687431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.687441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.687512 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.688381 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.689181 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.690792 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691092 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.691135 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.695122 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.695124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.695124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.695133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.698095 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.701644 (+ 3549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.701645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.701645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.701697 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.702658 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.702659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.702659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.702944 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.702989 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.703199 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.703274 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:59:10.703883 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.686326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:52565)
0928 08:59:10.686548 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.686560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.687169 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.687540 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.687541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.687953 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:10.688255 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.688262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.689316 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.689317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.690693 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.690695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.691225 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.691226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.691364 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.692253 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:10.692258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:10.694932 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:10.700996 (+ 6064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.700997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.700997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.701566 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.702079 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:10.702080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:10.702081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:10.702146 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:10.703041 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:10.703041 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:10.703084 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.703680 (+ 596us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.703784 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":98,"thread_start_us":103,"threads_started":1}
I20260928 08:59:10.704272 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36763
I20260928 08:59:10.704311 7723 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:10.704366 7723 heartbeater.cc:511] Master 127.0.98.62:36763 requested a full tablet report, sending...
I20260928 08:59:10.704821 6485 ts_manager.cc:194] Re-registered known tserver with Master: c7968924dbfe4876a603b11c09f3be2f (127.0.98.3:43215)
I20260928 08:59:10.705284 6485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52565
I20260928 08:59:10.703311 6486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:59:10.705636 6486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:10.705794 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.694291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.0.98.60:46771)
0928 08:59:10.694545 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.694548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.694728 (+ 180us) server_negotiation.cc:407] Connection header received
0928 08:59:10.694751 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.694751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.694764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.694824 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.695719 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.696519 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.699138 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.699435 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.699512 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.701980 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.701982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.701982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.701990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.704240 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.704627 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.704656 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.704656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.704721 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.704960 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.704961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.704961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.705126 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.705230 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.705593 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.705686 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":54,"threads_started":1}
I20260928 08:59:10.711269 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.694503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.0.98.61:43835)
0928 08:59:10.696586 (+ 2083us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.696588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.696597 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:10.696621 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.696622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.696635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.696790 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.697447 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.700171 (+ 2724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.702254 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.703423 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.703503 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.706878 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.706879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.706879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.706886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.709305 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.709840 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.709842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.709842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.709911 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.710616 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.710617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.710618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.710770 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.710897 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.711055 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.711148 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2040,"thread_start_us":57,"threads_started":1}
I20260928 08:59:10.714212 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.709706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.0.98.3:43215)
0928 08:59:10.710694 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.710699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.710709 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:10.710748 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.710750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.710768 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.710997 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:10.711050 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.712178 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.713324 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.713837 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.713942 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.714024 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.714098 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.714136 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":104,"threads_started":1}
I20260928 08:59:10.714200 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.709625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43215 (local address 127.0.0.1:55808)
0928 08:59:10.709713 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.709725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.709765 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.710871 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.710872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.710873 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:10.710921 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.710922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.712270 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.712271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.713219 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.713222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.713943 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.713944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.714012 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.714017 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.714084 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.714129 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 08:59:10.714864 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.710047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39547 (local address 127.0.0.1:54064)
0928 08:59:10.710379 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.710388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.710580 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.711135 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.711136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.711138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:10.711177 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.711179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.712554 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.712555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.713733 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.713736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.714643 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.714644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.714710 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.714713 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.714742 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.714782 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260928 08:59:10.714955 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.709833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35911 (local address 127.0.0.1:59570)
0928 08:59:10.710028 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.710039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.710535 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.711344 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.711345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.711347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:10.711383 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.711384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.712669 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.712670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.713734 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.713736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.714710 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.714711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.714765 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.714767 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.714818 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.714848 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":5,"thread_start_us":84,"threads_started":1}
I20260928 08:59:10.715296 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.710255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.98.1:35911)
0928 08:59:10.710701 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.710705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.710722 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:59:10.710754 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.710755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.710767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.710857 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:10.711470 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.712504 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.713836 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.714535 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.714585 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.714658 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.714850 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.714886 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":61,"threads_started":1}
I20260928 08:59:10.715410 7656 tablet_service.cc:1467] Processing CreateTablet for tablet 82cec79a82e74eacb5516a34aa8c450b (DEFAULT_TABLE table=test-table [id=1d4454622a7146529661e4323efe62f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:10.715773 7656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cec79a82e74eacb5516a34aa8c450b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:10.717103 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.710160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.98.2:39547)
0928 08:59:10.710978 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.710982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.710991 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:10.711027 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.711029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.711045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.711285 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:10.711346 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.712437 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.713834 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.714541 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.716857 (+ 2316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.716938 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.716964 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.717010 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":394,"threads_started":1}
I20260928 08:59:10.717399 7383 tablet_service.cc:1467] Processing CreateTablet for tablet 82cec79a82e74eacb5516a34aa8c450b (DEFAULT_TABLE table=test-table [id=1d4454622a7146529661e4323efe62f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:10.717752 7383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cec79a82e74eacb5516a34aa8c450b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:10.717782 7520 tablet_service.cc:1467] Processing CreateTablet for tablet 82cec79a82e74eacb5516a34aa8c450b (DEFAULT_TABLE table=test-table [id=1d4454622a7146529661e4323efe62f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:10.718089 7520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cec79a82e74eacb5516a34aa8c450b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:10.718959 7748 tablet_bootstrap.cc:492] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Bootstrap starting.
I20260928 08:59:10.719569 7748 tablet_bootstrap.cc:654] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:10.719868 7748 log.cc:824] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:10.720568 7748 tablet_bootstrap.cc:492] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: No bootstrap required, opened a new log
I20260928 08:59:10.720669 7748 ts_tablet_manager.cc:1397] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:10.721683 7750 tablet_bootstrap.cc:492] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Bootstrap starting.
I20260928 08:59:10.721777 7751 tablet_bootstrap.cc:492] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Bootstrap starting.
I20260928 08:59:10.722275 7750 tablet_bootstrap.cc:654] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:10.722352 7751 tablet_bootstrap.cc:654] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:10.722631 7750 log.cc:824] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:10.722615 7751 log.cc:824] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:10.723424 7750 tablet_bootstrap.cc:492] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: No bootstrap required, opened a new log
I20260928 08:59:10.723502 7750 ts_tablet_manager.cc:1397] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:59:10.723632 7748 raft_consensus.cc:348] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.723757 7748 raft_consensus.cc:374] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:10.723778 7748 raft_consensus.cc:729] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7968924dbfe4876a603b11c09f3be2f, State: Initialized, Role: FOLLOWER
I20260928 08:59:10.723912 7748 consensus_queue.cc:260] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.724256 7748 ts_tablet_manager.cc:1428] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 08:59:10.724268 7723 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260928 08:59:10.725885 7750 raft_consensus.cc:348] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.726002 7750 raft_consensus.cc:374] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:10.726022 7750 raft_consensus.cc:729] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08db1b76e35e496a98da7649d4a35f0d, State: Initialized, Role: FOLLOWER
I20260928 08:59:10.726135 7750 consensus_queue.cc:260] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.726372 7750 ts_tablet_manager.cc:1428] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:59:10.726420 7450 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260928 08:59:10.727488 7751 tablet_bootstrap.cc:492] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: No bootstrap required, opened a new log
I20260928 08:59:10.727567 7751 ts_tablet_manager.cc:1397] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 08:59:10.729568 7751 raft_consensus.cc:348] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.729681 7751 raft_consensus.cc:374] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:10.729702 7751 raft_consensus.cc:729] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4908d193378042b4b66566b3d42c99c7, State: Initialized, Role: FOLLOWER
I20260928 08:59:10.729802 7751 consensus_queue.cc:260] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.730008 7751 ts_tablet_manager.cc:1428] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 08:59:10.730093 7587 heartbeater.cc:503] Master 127.0.98.62:36763 was elected leader, sending a full tablet report...
I20260928 08:59:10.742173 7755 raft_consensus.cc:482] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:10.742316 7755 raft_consensus.cc:504] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.742697 7755 leader_election.cc:288] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7968924dbfe4876a603b11c09f3be2f (127.0.98.3:43215), 4908d193378042b4b66566b3d42c99c7 (127.0.98.2:39547)
I20260928 08:59:10.746660 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.742959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55809 (local address 127.0.98.2:39547)
0928 08:59:10.743023 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.743025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.743033 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:10.743060 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.743061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.743072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.743182 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:10.743347 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.744151 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.745682 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.746284 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.746358 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.746369 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.746545 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.746586 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:59:10.746661 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.742971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50861 (local address 127.0.98.3:43215)
0928 08:59:10.743017 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.743020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.743030 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:10.743060 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.743061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.743072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.743138 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:10.743347 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.744096 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.745683 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.746277 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.746358 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.746369 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.746544 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.746586 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 08:59:10.746677 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.742896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43215 (local address 127.0.98.1:50861)
0928 08:59:10.742946 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.742959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.743003 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.743143 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.743144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.743145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:10.743228 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.743230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.744199 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.744200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.745574 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.745576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.746376 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.746377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.746457 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.746461 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.746525 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.746575 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":65}
I20260928 08:59:10.746959 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.742866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39547 (local address 127.0.98.1:55809)
0928 08:59:10.742946 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:10.742967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:10.743003 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:10.743202 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:10.743203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:10.743204 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:10.743245 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.743246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.744219 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.744219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.745574 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:10.745576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.746376 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.746377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:10.746460 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.746461 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:59:10.746525 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:59:10.746575 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":58}
I20260928 08:59:10.747071 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cec79a82e74eacb5516a34aa8c450b" candidate_uuid: "08db1b76e35e496a98da7649d4a35f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7968924dbfe4876a603b11c09f3be2f" is_pre_election: true
I20260928 08:59:10.747231 7676 raft_consensus.cc:2454] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08db1b76e35e496a98da7649d4a35f0d in term 0.
I20260928 08:59:10.747308 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cec79a82e74eacb5516a34aa8c450b" candidate_uuid: "08db1b76e35e496a98da7649d4a35f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4908d193378042b4b66566b3d42c99c7" is_pre_election: true
I20260928 08:59:10.747450 7540 raft_consensus.cc:2454] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08db1b76e35e496a98da7649d4a35f0d in term 0.
I20260928 08:59:10.747475 7337 leader_election.cc:302] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08db1b76e35e496a98da7649d4a35f0d, c7968924dbfe4876a603b11c09f3be2f; no voters:
I20260928 08:59:10.747646 7755 raft_consensus.cc:2781] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:10.747702 7755 raft_consensus.cc:482] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:10.747730 7755 raft_consensus.cc:3037] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:10.748399 7755 raft_consensus.cc:504] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.748541 7755 leader_election.cc:288] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [CANDIDATE]: Term 1 election: Requested vote from peers c7968924dbfe4876a603b11c09f3be2f (127.0.98.3:43215), 4908d193378042b4b66566b3d42c99c7 (127.0.98.2:39547)
I20260928 08:59:10.748700 7676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cec79a82e74eacb5516a34aa8c450b" candidate_uuid: "08db1b76e35e496a98da7649d4a35f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7968924dbfe4876a603b11c09f3be2f"
I20260928 08:59:10.748776 7676 raft_consensus.cc:3037] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:10.748824 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cec79a82e74eacb5516a34aa8c450b" candidate_uuid: "08db1b76e35e496a98da7649d4a35f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4908d193378042b4b66566b3d42c99c7"
I20260928 08:59:10.748890 7540 raft_consensus.cc:3037] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:10.749658 7540 raft_consensus.cc:2454] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08db1b76e35e496a98da7649d4a35f0d in term 1.
I20260928 08:59:10.749876 7676 raft_consensus.cc:2454] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08db1b76e35e496a98da7649d4a35f0d in term 1.
I20260928 08:59:10.749866 7339 leader_election.cc:302] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08db1b76e35e496a98da7649d4a35f0d, 4908d193378042b4b66566b3d42c99c7; no voters:
I20260928 08:59:10.749948 7755 raft_consensus.cc:2781] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:10.750102 7755 raft_consensus.cc:686] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 1 LEADER]: Becoming Leader. State: Replica: 08db1b76e35e496a98da7649d4a35f0d, State: Running, Role: LEADER
I20260928 08:59:10.750211 7755 consensus_queue.cc:237] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } }
I20260928 08:59:10.751011 6486 catalog_manager.cc:5495] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 08db1b76e35e496a98da7649d4a35f0d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "08db1b76e35e496a98da7649d4a35f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08db1b76e35e496a98da7649d4a35f0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:10.763845 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.761468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.0.98.1:35911)
0928 08:59:10.761532 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.761535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.761544 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:10.761613 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.761615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.761634 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.761693 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:10.761826 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.762340 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.763169 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.763343 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.763392 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.763522 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:10.763622 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:10.763681 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.763717 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.763733 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:59:10.765749 7676 raft_consensus.cc:1261] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 1 FOLLOWER]: Refusing update from remote peer 08db1b76e35e496a98da7649d4a35f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:10.765774 7540 raft_consensus.cc:1261] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 1 FOLLOWER]: Refusing update from remote peer 08db1b76e35e496a98da7649d4a35f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:10.766099 7757 consensus_queue.cc:1035] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7968924dbfe4876a603b11c09f3be2f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:10.766224 7755 consensus_queue.cc:1035] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4908d193378042b4b66566b3d42c99c7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:10.768800 7760 mvcc.cc:205] Tried to move back new op lower bound from 7334240054333497344 to 7334240054273286144. Current Snapshot: MvccSnapshot[applied={T|T < 7334240054333497344}]
I20260928 08:59:10.769007 7762 mvcc.cc:205] Tried to move back new op lower bound from 7334240054333497344 to 7334240054273286144. Current Snapshot: MvccSnapshot[applied={T|T < 7334240054333497344}]
I20260928 08:59:10.769294 7761 mvcc.cc:205] Tried to move back new op lower bound from 7334240054333497344 to 7334240054273286144. Current Snapshot: MvccSnapshot[applied={T|T < 7334240054333497344}]
I20260928 08:59:10.771806 6485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:10.771917 6485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:10.773221 6485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2dbcf189fc949769c80262bc045854d: Sending DeleteTablet for 3 replicas of tablet 82cec79a82e74eacb5516a34aa8c450b
I20260928 08:59:10.773571 7383 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cec79a82e74eacb5516a34aa8c450b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59570
I20260928 08:59:10.773633 7520 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cec79a82e74eacb5516a34aa8c450b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54064
I20260928 08:59:10.773802 7656 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cec79a82e74eacb5516a34aa8c450b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55808
I20260928 08:59:10.773836 7772 tablet_replica.cc:330] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: stopping tablet replica
I20260928 08:59:10.773936 7772 raft_consensus.cc:2229] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:10.774053 7772 raft_consensus.cc:2258] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:10.774045 7773 tablet_replica.cc:330] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: stopping tablet replica
I20260928 08:59:10.774204 7773 raft_consensus.cc:2229] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:10.774344 7773 raft_consensus.cc:2258] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:10.774523 7772 ts_tablet_manager.cc:1905] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:10.774865 7773 ts_tablet_manager.cc:1905] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:10.775305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6454
I20260928 08:59:10.776461 7772 ts_tablet_manager.cc:1918] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:10.776540 7772 log.cc:1196] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/82cec79a82e74eacb5516a34aa8c450b
I20260928 08:59:10.776717 7773 ts_tablet_manager.cc:1918] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:10.776851 7772 ts_tablet_manager.cc:1939] T 82cec79a82e74eacb5516a34aa8c450b P 4908d193378042b4b66566b3d42c99c7: Deleting consensus metadata
I20260928 08:59:10.776859 7773 log.cc:1196] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/82cec79a82e74eacb5516a34aa8c450b
I20260928 08:59:10.777206 7773 ts_tablet_manager.cc:1939] T 82cec79a82e74eacb5516a34aa8c450b P c7968924dbfe4876a603b11c09f3be2f: Deleting consensus metadata
I20260928 08:59:10.780802 7771 tablet_replica.cc:330] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: stopping tablet replica
I20260928 08:59:10.780967 7771 raft_consensus.cc:2229] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:59:10.781157 7771 raft_consensus.cc:2258] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:10.781692 7771 ts_tablet_manager.cc:1905] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 08:59:10.781937 7475 connection.cc:535] server connection from 127.0.0.1:54064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 08:59:10.782114 7610 connection.cc:535] server connection from 127.0.0.1:55808 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:59:10.783520 7771 ts_tablet_manager.cc:1918] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:10.783658 7771 log.cc:1196] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/82cec79a82e74eacb5516a34aa8c450b
I20260928 08:59:10.783968 7771 ts_tablet_manager.cc:1939] T 82cec79a82e74eacb5516a34aa8c450b P 08db1b76e35e496a98da7649d4a35f0d: Deleting consensus metadata
W20260928 08:59:10.784358 7338 connection.cc:441] server connection from 127.0.0.1:59570 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59570 (request call id 1) could send its response
I20260928 08:59:10.789932 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.774658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.98.60:46771)
0928 08:59:10.774700 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.774702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.780860 (+ 6158us) server_negotiation.cc:407] Connection header received
0928 08:59:10.780964 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.780965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.780980 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.781010 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.782118 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.782917 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.784202 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.784519 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.784566 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.786234 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.786236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.786236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.786243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.787945 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.788919 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.788921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.788921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.788978 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.789234 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.789235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.789235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.789380 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.789468 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.789676 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.789802 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:10.794449 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.784771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.0.98.60:46771)
0928 08:59:10.784809 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.784811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.785008 (+ 197us) server_negotiation.cc:407] Connection header received
0928 08:59:10.785116 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.785117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.785127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.785155 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.785897 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.786608 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.787783 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.788107 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.788145 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.789657 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.789658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.789658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.789664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.792460 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.793129 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.793130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.793131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.793212 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.793530 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.793531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.793532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.793871 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.793958 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.794267 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.794346 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:10.795010 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:10.784448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.0.98.61:43835)
0928 08:59:10.784489 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:10.784492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:10.784871 (+ 379us) server_negotiation.cc:407] Connection header received
0928 08:59:10.784971 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:10.784972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:10.784985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:10.785014 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:10.785762 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.786526 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.787592 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:10.787900 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:10.787944 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:10.789579 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:10.789581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:10.789581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:10.789588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:10.793297 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.793940 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.793942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.793942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.794006 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:10.794402 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:10.794403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:10.794403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:10.794534 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:10.794615 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:10.794828 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 08:59:10.794918 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 08:59:11.769943 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:11.769661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:56921)
0928 08:59:11.769807 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:11.769885 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260928 08:59:11.770025 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:11.769670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:55813)
0928 08:59:11.769797 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:11.769907 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
W20260928 08:59:11.770274 7450 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:11.770463 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:11.770159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:52773)
0928 08:59:11.770360 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:11.770425 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":57,"threads_started":1}
W20260928 08:59:11.770470 7587 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
W20260928 08:59:11.770594 7723 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36763 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:12.274083 7784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2dbcf189fc949769c80262bc045854d)
I20260928 08:59:12.274245 7784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:12.274518 7784 leader_election.cc:288] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763)
I20260928 08:59:12.274801 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:12.274629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:45536)
0928 08:59:12.274713 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:12.274759 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":50,"threads_started":1}
W20260928 08:59:12.274926 7070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:12.275295 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:12.275200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:45548)
0928 08:59:12.275250 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:12.275273 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 08:59:12.275393 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:12.275441 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5735da49e7e440ec9b55cf38854a6a30; no voters: f2dbcf189fc949769c80262bc045854d
I20260928 08:59:12.275514 7784 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:59:12.771133 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:12.770813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:37393)
0928 08:59:12.771022 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:12.771065 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
I20260928 08:59:12.771430 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:12.770856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:33911)
0928 08:59:12.771208 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:12.771254 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":62,"threads_started":1}
I20260928 08:59:12.771557 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:12.771056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:42999)
0928 08:59:12.771490 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:12.771522 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":53,"threads_started":1}
I20260928 08:59:13.772125 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:13.771719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:45675)
0928 08:59:13.772024 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:13.772067 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":61,"threads_started":1}
I20260928 08:59:13.772161 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:13.771957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:46905)
0928 08:59:13.772051 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:13.772119 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":43,"threads_started":1}
I20260928 08:59:13.772214 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:13.771837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:35437)
0928 08:59:13.772121 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:13.772161 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":59,"threads_started":1}
I20260928 08:59:13.910223 7793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2dbcf189fc949769c80262bc045854d)
I20260928 08:59:13.910377 7793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:13.910684 7793 leader_election.cc:288] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763)
I20260928 08:59:13.911047 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:13.910857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:45740)
0928 08:59:13.910961 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:13.911003 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260928 08:59:13.911485 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:13.911372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:45756)
0928 08:59:13.911436 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:13.911462 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:59:13.911564 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:13.911613 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5735da49e7e440ec9b55cf38854a6a30; no voters: f2dbcf189fc949769c80262bc045854d
I20260928 08:59:13.911667 7793 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:59:14.772940 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:14.772670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:33005)
0928 08:59:14.772836 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:14.772881 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260928 08:59:14.773530 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:14.772757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:47461)
0928 08:59:14.773459 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:14.773490 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":78,"threads_started":1}
I20260928 08:59:14.773531 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:14.772668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:57693)
0928 08:59:14.773451 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:14.773489 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":69,"threads_started":1}
I20260928 08:59:15.774410 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:15.774182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:40129)
0928 08:59:15.774349 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:15.774376 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":54,"threads_started":1}
I20260928 08:59:15.774407 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:15.773820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:46725)
0928 08:59:15.774307 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:15.774351 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":65,"threads_started":1}
I20260928 08:59:15.774688 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:15.773930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:53991)
0928 08:59:15.774560 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:15.774633 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":46,"threads_started":1}
I20260928 08:59:15.932920 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2dbcf189fc949769c80262bc045854d)
I20260928 08:59:15.933022 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:15.933235 7802 leader_election.cc:288] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763)
I20260928 08:59:15.933614 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:15.933398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:45978)
0928 08:59:15.933513 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:15.933555 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20260928 08:59:15.934051 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:15.933929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:45980)
0928 08:59:15.933988 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:15.934026 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 08:59:15.934135 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:15.934201 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5735da49e7e440ec9b55cf38854a6a30; no voters: f2dbcf189fc949769c80262bc045854d
I20260928 08:59:15.934296 7802 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:59:16.775295 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:16.775068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:52743)
0928 08:59:16.775184 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:16.775234 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 08:59:16.775586 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:16.774860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:38271)
0928 08:59:16.775502 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:16.775551 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":67,"threads_started":1}
I20260928 08:59:16.775655 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:16.774860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:48233)
0928 08:59:16.775529 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:16.775619 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":66,"threads_started":1}
I20260928 08:59:17.776173 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:17.775710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:40037)
0928 08:59:17.776064 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:17.776111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":78,"threads_started":1}
I20260928 08:59:17.776368 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:17.776101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:57387)
0928 08:59:17.776298 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:17.776330 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":55,"threads_started":1}
I20260928 08:59:17.776449 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:17.776089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:33777)
0928 08:59:17.776339 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:17.776392 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":55,"threads_started":1}
I20260928 08:59:18.777133 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:18.776775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:33073)
0928 08:59:18.777031 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:18.777076 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
I20260928 08:59:18.777236 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:18.776890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:58925)
0928 08:59:18.777169 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:18.777201 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":46,"threads_started":1}
I20260928 08:59:18.777379 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:18.777018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:57547)
0928 08:59:18.777283 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:18.777324 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":58,"threads_started":1}
I20260928 08:59:19.606947 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2dbcf189fc949769c80262bc045854d)
I20260928 08:59:19.607040 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:19.607281 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763)
I20260928 08:59:19.607743 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:19.607531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:46280)
0928 08:59:19.607666 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:19.607705 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
W20260928 08:59:19.607834 7070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 08:59:19.608211 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:19.608113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:46282)
0928 08:59:19.608166 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:19.608193 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 08:59:19.608290 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:19.608371 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5735da49e7e440ec9b55cf38854a6a30; no voters: f2dbcf189fc949769c80262bc045854d
I20260928 08:59:19.608472 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:59:19.778172 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:19.777912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:47135)
0928 08:59:19.778050 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:19.778103 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260928 08:59:19.778302 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:19.777862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:42955)
0928 08:59:19.778212 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:19.778250 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":66,"threads_started":1}
I20260928 08:59:19.778350 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:19.778037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:42003)
0928 08:59:19.778285 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:19.778317 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":43,"threads_started":1}
I20260928 08:59:20.778997 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:20.778741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:57321)
0928 08:59:20.778888 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:20.778933 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 08:59:20.779179 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:20.778864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:56333)
0928 08:59:20.779091 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:20.779128 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":54,"threads_started":1}
I20260928 08:59:20.779279 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:20.778864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:50563)
0928 08:59:20.779189 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:20.779223 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":57,"threads_started":1}
I20260928 08:59:21.779873 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:21.779611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:53731)
0928 08:59:21.779770 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:21.779812 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 08:59:21.779999 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:21.779689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:55595)
0928 08:59:21.779927 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:21.779962 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":56,"threads_started":1}
I20260928 08:59:21.779994 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:21.779783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:34369)
0928 08:59:21.779912 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:21.779948 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":42,"threads_started":1}
I20260928 08:59:22.780700 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:22.780448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:56387)
0928 08:59:22.780579 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:22.780620 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 08:59:22.780855 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:22.780673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:43617)
0928 08:59:22.780783 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:22.780816 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":46,"threads_started":1}
I20260928 08:59:22.780975 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:22.780789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:49339)
0928 08:59:22.780894 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:22.780928 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260928 08:59:23.781488 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:23.781131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:32831)
0928 08:59:23.781388 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:23.781430 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
I20260928 08:59:23.781814 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:23.781310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:56747)
0928 08:59:23.781725 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:23.781765 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":52,"threads_started":1}
I20260928 08:59:23.781986 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:23.781761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:37305)
0928 08:59:23.781899 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:23.781939 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 08:59:24.237437 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2dbcf189fc949769c80262bc045854d)
I20260928 08:59:24.237542 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:24.237718 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763)
I20260928 08:59:24.238153 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:24.237929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52014)
0928 08:59:24.238066 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:24.238114 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260928 08:59:24.238653 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:24.238498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:52018)
0928 08:59:24.238552 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:24.238634 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 08:59:24.238739 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:24.238818 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5735da49e7e440ec9b55cf38854a6a30; no voters: f2dbcf189fc949769c80262bc045854d
I20260928 08:59:24.238924 7832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:59:24.782495 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:24.781993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:36927)
0928 08:59:24.782383 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:24.782431 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":67,"threads_started":1}
I20260928 08:59:24.782590 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:24.782365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:37933)
0928 08:59:24.782494 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:24.782538 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260928 08:59:24.782853 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:24.782553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:36089)
0928 08:59:24.782772 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:24.782808 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":53,"threads_started":1}
I20260928 08:59:25.783365 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:25.783123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:57825)
0928 08:59:25.783267 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:25.783310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260928 08:59:25.783572 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:25.783258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:52805)
0928 08:59:25.783500 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:25.783534 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":43,"threads_started":1}
I20260928 08:59:25.783704 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:25.782996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:41857)
0928 08:59:25.783613 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:25.783653 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":69,"threads_started":1}
I20260928 08:59:26.784207 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:26.783864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:38201)
0928 08:59:26.784107 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:26.784160 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":67,"threads_started":1}
I20260928 08:59:26.784560 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:26.784136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:48085)
0928 08:59:26.784472 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:26.784508 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":53,"threads_started":1}
I20260928 08:59:26.784678 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:26.784188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:48941)
0928 08:59:26.784560 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:26.784595 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":40,"threads_started":1}
I20260928 08:59:27.785305 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:27.785063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:33201)
0928 08:59:27.785208 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:27.785249 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260928 08:59:27.785362 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:27.784828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:51143)
0928 08:59:27.785269 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:27.785311 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":69,"threads_started":1}
I20260928 08:59:27.785665 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:27.785012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:36741)
0928 08:59:27.785562 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:27.785607 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":55,"threads_started":1}
I20260928 08:59:28.786280 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:28.785921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:51923)
0928 08:59:28.786170 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:28.786219 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":63,"threads_started":1}
I20260928 08:59:28.786311 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:28.785921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:55951)
0928 08:59:28.786215 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:28.786257 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":61,"threads_started":1}
I20260928 08:59:28.786424 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:28.786225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:58203)
0928 08:59:28.786347 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:28.786379 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 08:59:29.787117 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:29.786884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:53407)
0928 08:59:29.787014 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:29.787058 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 08:59:29.787118 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:29.786883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:51941)
0928 08:59:29.787048 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:29.787082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260928 08:59:29.787401 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:29.787150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:53223)
0928 08:59:29.787303 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:29.787343 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260928 08:59:30.787941 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:30.787640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:38373)
0928 08:59:30.787840 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:30.787885 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260928 08:59:30.788066 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:30.787806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:35425)
0928 08:59:30.787925 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:30.788014 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":52,"threads_started":1}
I20260928 08:59:30.788359 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:30.787893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:36035)
0928 08:59:30.788287 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:30.788325 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":63,"threads_started":1}
I20260928 08:59:31.788766 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:31.788506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:50541)
0928 08:59:31.788663 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:31.788708 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260928 08:59:31.788798 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:31.788474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:60345)
0928 08:59:31.788679 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:31.788744 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260928 08:59:31.789304 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:31.789086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:36211)
0928 08:59:31.789219 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:31.789255 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260928 08:59:32.789448 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:32.789213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:45925)
0928 08:59:32.789349 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:32.789391 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 08:59:32.789670 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:32.789213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:34261)
0928 08:59:32.789590 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:32.789629 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":67,"threads_started":1}
I20260928 08:59:32.789971 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:32.789752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:36265)
0928 08:59:32.789887 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:32.789925 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260928 08:59:33.790593 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:33.790155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:58553)
0928 08:59:33.790476 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:33.790532 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":64,"threads_started":1}
I20260928 08:59:33.790627 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:33.790427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:49447)
0928 08:59:33.790540 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:33.790575 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":46,"threads_started":1}
I20260928 08:59:33.791043 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:33.790812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:56143)
0928 08:59:33.790960 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:33.790996 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":51,"threads_started":1}
I20260928 08:59:34.297050 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f2dbcf189fc949769c80262bc045854d)
I20260928 08:59:34.297156 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2dbcf189fc949769c80262bc045854d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36763 } } peers { permanent_uuid: "5735da49e7e440ec9b55cf38854a6a30" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43835 } attrs { promote: false } } peers { permanent_uuid: "4e21e8e13c084226b372376a184c8dd7" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46771 } attrs { promote: true } }
I20260928 08:59:34.297317 7865 leader_election.cc:288] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763)
I20260928 08:59:34.297648 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:34.297469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:34370)
0928 08:59:34.297568 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:34.297610 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
W20260928 08:59:34.297773 7070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 08:59:34.298214 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:34.298103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.0.1:34386)
0928 08:59:34.298158 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:34.298192 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 08:59:34.298347 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2dbcf189fc949769c80262bc045854d (127.0.98.62:36763): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
I20260928 08:59:34.298400 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 5735da49e7e440ec9b55cf38854a6a30; no voters: f2dbcf189fc949769c80262bc045854d
I20260928 08:59:34.298462 7865 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5735da49e7e440ec9b55cf38854a6a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:59:34.791455 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:34.791216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:44571)
0928 08:59:34.791354 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:34.791394 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260928 08:59:34.791541 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:34.791216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:57893)
0928 08:59:34.791444 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:34.791486 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":66,"threads_started":1}
I20260928 08:59:34.791741 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:34.791560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:59945)
0928 08:59:34.791677 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:34.791706 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
I20260928 08:59:35.792270 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:35.791970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:47847)
0928 08:59:35.792140 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:35.792218 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260928 08:59:35.792266 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:35.792029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:38557)
0928 08:59:35.792151 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:35.792202 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":45,"threads_started":1}
I20260928 08:59:35.792555 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:35.792305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:60329)
0928 08:59:35.792422 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:35.792479 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":42,"threads_started":1}
I20260928 08:59:36.793049 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:36.792837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:35463)
0928 08:59:36.792968 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:36.792995 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260928 08:59:36.793105 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:36.792834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:35809)
0928 08:59:36.792958 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:36.792995 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 08:59:36.793263 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:36.792988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:59099)
0928 08:59:36.793174 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:36.793214 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":51,"threads_started":1}
I20260928 08:59:37.793818 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:37.793584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:42449)
0928 08:59:37.793716 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:37.793757 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260928 08:59:37.793892 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:37.793697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:35003)
0928 08:59:37.793813 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:37.793847 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":45,"threads_started":1}
I20260928 08:59:37.793892 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:37.793667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:43793)
0928 08:59:37.793831 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:37.793857 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260928 08:59:38.794708 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:38.794245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:54465)
0928 08:59:38.794559 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:38.794603 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":65,"threads_started":1}
I20260928 08:59:38.794719 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:38.794511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:60069)
0928 08:59:38.794630 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:38.794668 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 08:59:38.795285 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:38.795005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:60271)
0928 08:59:38.795182 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:38.795236 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 08:59:39.795419 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:39.795163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:50247)
0928 08:59:39.795285 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:39.795338 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260928 08:59:39.795483 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:39.795163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:54729)
0928 08:59:39.795304 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:39.795362 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 08:59:39.795912 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:39.795722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:55067)
0928 08:59:39.795840 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:39.795881 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 08:59:40.796178 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:40.795937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.3:45919)
0928 08:59:40.796101 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:40.796130 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260928 08:59:40.796216 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:40.795925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.1:42833)
0928 08:59:40.796076 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:40.796119 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
I20260928 08:59:40.796429 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:40.796236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36763 (local address 127.0.98.2:49845)
0928 08:59:40.796346 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:40.796381 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:36763,127.0.98.61:43835,127.0.98.60:46771, num_attempts: 1)
I20260928 08:59:40.974579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 7231
I20260928 08:59:40.980512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 7457
I20260928 08:59:40.985083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 7593
I20260928 08:59:40.989403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 6950
I20260928 08:59:40.994004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 7137
2026-09-28T08:59:40Z chronyd exiting
I20260928 08:59:41.009932 392 test_util.cc:175] -----------------------------------------------
I20260928 08:59:41.009999 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585920139418-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33082 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:41 dist-test-slave-286n krb5kdc[7891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:41 dist-test-slave-286n krb5kdc[7891](info): set up 2 sockets
Sep 28 08:59:41 dist-test-slave-286n krb5kdc[7891](info): commencing operation
krb5kdc: starting...
W20260928 08:59:42.029206 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 08:59:42.152108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:42.235563 7904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:42.235834 7904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:42.235874 7904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:42.237200 7904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:42.237265 7904 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:59:42.237289 7904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:42.237322 7904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:42.237347 7904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:42.237373 7904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:42.238921 7904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42051
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:42.239209 7904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:42.239442 7904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:59:42.244769 7904 server_base.cc:1034] running on GCE node
W20260928 08:59:42.244807 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.244891 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.244904 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:42.245258 7904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:42.245518 7904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:42.246659 7904 hybrid_clock.cc:648] HybridClock initialized: now 1790585982246643 us; error 35 us; skew 500 ppm
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:42.249104 7904 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 08:59:42.249578 7904 webserver.cc:460] Webserver started at http://127.0.98.62:42549/ using document root <none> and password file <none>
I20260928 08:59:42.249823 7904 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:42.249877 7904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:42.250003 7904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:42.251093 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "7e6fdb5c6d2949079f243b1df5434e49"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "6d32937d6f0782a97a22100f092af0fc"
server_key_iv: "dad1ab5944311a449c3366c0978ceadc"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.251456 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e6fdb5c6d2949079f243b1df5434e49"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "6d32937d6f0782a97a22100f092af0fc"
server_key_iv: "dad1ab5944311a449c3366c0978ceadc"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.252821 7904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 08:59:42.253604 7919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.253845 7904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:42.253950 7904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "7e6fdb5c6d2949079f243b1df5434e49"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "6d32937d6f0782a97a22100f092af0fc"
server_key_iv: "dad1ab5944311a449c3366c0978ceadc"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.254026 7904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:42.270275 7904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:42.272336 7904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:42.272511 7904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:42.277254 7904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42051
I20260928 08:59:42.277300 7981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42051 every 8 connection(s)
I20260928 08:59:42.277631 7904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:42.278268 7982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:42.280427 7982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Bootstrap starting.
I20260928 08:59:42.281029 7982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:42.281261 7982 log.cc:824] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:42.281894 7982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: No bootstrap required, opened a new log
I20260928 08:59:42.283867 7982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdb5c6d2949079f243b1df5434e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260928 08:59:42.284013 7982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:42.284035 7982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6fdb5c6d2949079f243b1df5434e49, State: Initialized, Role: FOLLOWER
I20260928 08:59:42.284135 7982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdb5c6d2949079f243b1df5434e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260928 08:59:42.284219 7982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:42.284256 7982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:42.284305 7982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:42.284962 7982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdb5c6d2949079f243b1df5434e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260928 08:59:42.285095 7982 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6fdb5c6d2949079f243b1df5434e49; no voters:
I20260928 08:59:42.285303 7982 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:42.285418 7985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:42.285570 7982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:42.285604 7985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [term 1 LEADER]: Becoming Leader. State: Replica: 7e6fdb5c6d2949079f243b1df5434e49, State: Running, Role: LEADER
I20260928 08:59:42.285740 7985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdb5c6d2949079f243b1df5434e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } }
I20260928 08:59:42.286129 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6fdb5c6d2949079f243b1df5434e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdb5c6d2949079f243b1df5434e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } } }
I20260928 08:59:42.286137 7987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6fdb5c6d2949079f243b1df5434e49. Latest consensus state: current_term: 1 leader_uuid: "7e6fdb5c6d2949079f243b1df5434e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdb5c6d2949079f243b1df5434e49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42051 } } }
I20260928 08:59:42.286342 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:42.286370 7987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49 [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:42.286571 7994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:42.287370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 7904
I20260928 08:59:42.287484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:42.287695 392 external_mini_cluster.cc:1428] Setting key 4718b957452da88350083a252300dad6
I20260928 08:59:42.288136 7994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:42.289849 7994 catalog_manager.cc:1347] Generated new cluster ID: d3ee8df5129b4fa98190da5078d6ff72
I20260928 08:59:42.289907 7994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:42.300515 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.288245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.0.98.62:42051)
0928 08:59:42.288509 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.288520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.288552 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:59:42.288858 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.288867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.288944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.289880 (+ 936us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.289976 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.291412 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.292515 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.293190 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.293654 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.296252 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.296256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.296257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.296271 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.298340 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.298777 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.298781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.298782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.298910 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.299119 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.299119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.299120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.299445 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.300039 (+ 594us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.300113 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.300264 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260928 08:59:42.309154 7994 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:42.309836 7994 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:42.315805 7994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Generated new TSK 0
I20260928 08:59:42.316035 7994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 08:59:42.348083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:42.429016 8008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:42.429234 8008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:42.429257 8008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:42.430362 8008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:42.430395 8008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:42.430415 8008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:59:42.430433 8008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:42.431803 8008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:42.432046 8008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:42.432291 8008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:42.437328 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.437346 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.437346 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:42.437763 8008 server_base.cc:1034] running on GCE node
I20260928 08:59:42.437971 8008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:42.438211 8008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:42.439373 8008 hybrid_clock.cc:648] HybridClock initialized: now 1790585982439358 us; error 26 us; skew 500 ppm
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:42.446604 8008 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 08:59:42.447150 8008 webserver.cc:460] Webserver started at http://127.0.98.1:43225/ using document root <none> and password file <none>
I20260928 08:59:42.447395 8008 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:42.447463 8008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:42.447613 8008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:42.448788 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "e08bccd37dc4494081aca2849c570e0f"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "9c978bd613e5c1f0dee43aaf79a58389"
server_key_iv: "444204e4eb058c1eefa6edd792bc0452"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.449172 8008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e08bccd37dc4494081aca2849c570e0f"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "9c978bd613e5c1f0dee43aaf79a58389"
server_key_iv: "444204e4eb058c1eefa6edd792bc0452"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.450587 8008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:59:42.451393 8023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.451560 8008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:42.451671 8008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "e08bccd37dc4494081aca2849c570e0f"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "9c978bd613e5c1f0dee43aaf79a58389"
server_key_iv: "444204e4eb058c1eefa6edd792bc0452"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.451736 8008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:42.466467 8008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:42.468920 8008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:42.469077 8008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:42.469669 8008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:42.469728 8008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.469765 8008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:42.469794 8008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:42.477101 8008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37007
I20260928 08:59:42.477186 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37007 every 8 connection(s)
I20260928 08:59:42.477545 8008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:42.479331 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.470073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36983 (local address 127.0.98.62:42051)
0928 08:59:42.470151 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.470153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.470350 (+ 197us) server_negotiation.cc:407] Connection header received
0928 08:59:42.470840 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.470841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.470858 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.470948 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.471567 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.472416 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.473606 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.473828 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.473916 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.476483 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.476484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.476485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.476496 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.478285 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.478631 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.478632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.478632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.478685 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.478874 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.478875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.478875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.478972 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.479018 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.479200 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.479267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 08:59:42.479532 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.470064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.1:36983)
0928 08:59:42.470217 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.470257 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.470745 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.471000 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.471003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.471222 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:42.471466 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.471471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.472535 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.472536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.473513 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.473516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.473925 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.473927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.474050 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.474572 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:42.474581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:42.476386 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:42.478352 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.478357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.478359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.478564 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.478743 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.478743 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.478743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.478824 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.479008 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:42.479010 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:42.479114 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.479187 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.479299 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 08:59:42.480039 8139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260928 08:59:42.480121 8139 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:42.480301 8139 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260928 08:59:42.480875 7935 ts_manager.cc:194] Registered new tserver with Master: e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007)
I20260928 08:59:42.481527 7935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36983
I20260928 08:59:42.483171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8008
I20260928 08:59:42.483306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:42.483503 392 external_mini_cluster.cc:1428] Setting key b6bda1fc39cfebdaf4ce1085538fa9a3
I20260928 08:59:42.486888 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.481050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47215 (local address 127.0.98.62:42051)
0928 08:59:42.481093 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.481095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.481231 (+ 136us) server_negotiation.cc:407] Connection header received
0928 08:59:42.481301 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.481302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.481312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.481366 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.481666 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.482132 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.482832 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.483054 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.483128 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.484097 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.484099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.484099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.484105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.485618 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.485960 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.485961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.485961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.486035 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.486211 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.486212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.486213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.486363 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.486443 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.486731 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.486819 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:59:42.486960 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.481009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.1:47215)
0928 08:59:42.481147 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.481157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.481245 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.481373 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.481374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.481533 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:42.481588 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.481589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.482220 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.482221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.482752 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.482754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.483125 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.483125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.483191 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.483634 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:42.483639 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:42.484005 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:42.485706 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.485707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.485707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.485876 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.486084 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.486085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.486085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.486139 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.486451 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:42.486452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:42.486595 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.486704 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.486815 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 08:59:42.511135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:42.592154 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:42.592396 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:42.592437 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:42.593662 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:42.593741 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:42.593776 8148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:59:42.593819 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:42.595348 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:42.595614 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:42.595853 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:42.600946 8153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.600943 8156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.600984 8154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:42.601209 8148 server_base.cc:1034] running on GCE node
I20260928 08:59:42.601397 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:42.601581 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:42.602715 8148 hybrid_clock.cc:648] HybridClock initialized: now 1790585982602707 us; error 23 us; skew 500 ppm
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:42.605243 8148 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 08:59:42.605716 8148 webserver.cc:460] Webserver started at http://127.0.98.2:44601/ using document root <none> and password file <none>
I20260928 08:59:42.605960 8148 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:42.606030 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:42.606175 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:42.607231 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "3a2812d3364a6e2d7b9e97c0bccf53ca"
server_key_iv: "e459b79a7b839448132d4028c3521f60"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.607599 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "3a2812d3364a6e2d7b9e97c0bccf53ca"
server_key_iv: "e459b79a7b839448132d4028c3521f60"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.609009 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:42.609760 8163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.609939 8148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:42.610042 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "3a2812d3364a6e2d7b9e97c0bccf53ca"
server_key_iv: "e459b79a7b839448132d4028c3521f60"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.610114 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:42.630306 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:42.632696 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:42.632858 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:42.633416 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:42.633477 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.633522 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:42.633550 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:42.640806 8148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35995
I20260928 08:59:42.640892 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35995 every 8 connection(s)
I20260928 08:59:42.641188 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 08:59:42.643404 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.634010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46013 (local address 127.0.98.62:42051)
0928 08:59:42.634088 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.634090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.634354 (+ 264us) server_negotiation.cc:407] Connection header received
0928 08:59:42.635113 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.635132 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.635146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.635236 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.635939 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.636846 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.637855 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.638194 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.638269 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.640996 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.640998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.640999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.641006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.642350 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.642706 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.642706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.642707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.642755 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.642924 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.642924 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.642924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.643020 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.643068 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.643287 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.643344 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:59:42.643574 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.633984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.2:46013)
0928 08:59:42.634189 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.634239 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.634995 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.635345 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.635349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.635580 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:42.635782 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.635787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.636951 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.636952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.637747 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.637749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.638310 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.638311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.638481 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.638998 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:42.639008 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:42.640883 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:42.642423 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.642428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.642430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.642637 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.642796 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.642796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.642796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.642873 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.643074 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:42.643078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:42.643184 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.643264 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.643385 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260928 08:59:42.644013 8279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260928 08:59:42.644098 8279 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:42.644268 8279 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260928 08:59:42.644716 7936 ts_manager.cc:194] Registered new tserver with Master: 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995)
I20260928 08:59:42.645205 7936 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46013
I20260928 08:59:42.646162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8148
I20260928 08:59:42.646271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:42.646473 392 external_mini_cluster.cc:1428] Setting key 100238f91c60440751b4bdea96e579e0
I20260928 08:59:42.651008 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.645148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55387 (local address 127.0.98.62:42051)
0928 08:59:42.645200 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.645202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.645255 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:59:42.645365 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.645365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.645376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.645438 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.645708 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.646338 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.647279 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.647575 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.647648 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.648668 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.648669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.648670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.648676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.649935 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.650317 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.650317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.650318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.650356 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.650518 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.650519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.650519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.650657 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.650727 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.650900 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.650952 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:42.651111 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.645080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.2:55387)
0928 08:59:42.645195 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.645205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.645282 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.645440 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.645440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.645579 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:42.645634 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.645635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.646415 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.646416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.647193 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.647195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.647645 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.647646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.647699 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.648140 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:42.648145 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:42.648548 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:42.650013 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.650014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.650015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.650249 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.650398 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.650399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.650399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.650464 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.650737 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:42.650738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:42.650791 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.650878 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.650988 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 08:59:42.676630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42051
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:42.762148 8288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:42.762367 8288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:42.762389 8288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:42.763545 8288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:42.763588 8288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:42.763609 8288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:59:42.763667 8288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:42.765146 8288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:42.765408 8288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:42.765643 8288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:42.770655 8294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.770690 8293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:42.770655 8296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:42.770843 8288 server_base.cc:1034] running on GCE node
I20260928 08:59:42.771052 8288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:42.771334 8288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:42.772488 8288 hybrid_clock.cc:648] HybridClock initialized: now 1790585982772479 us; error 31 us; skew 500 ppm
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:42.775013 8288 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 08:59:42.775473 8288 webserver.cc:460] Webserver started at http://127.0.98.3:35135/ using document root <none> and password file <none>
I20260928 08:59:42.775712 8288 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:42.775777 8288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:42.775913 8288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:42.777047 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "1d5e22f0ff8ff257dfecdf0d7eade977"
server_key_iv: "9adc70a8468776ca8b9f386c70f7eac2"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.777412 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "1d5e22f0ff8ff257dfecdf0d7eade977"
server_key_iv: "9adc70a8468776ca8b9f386c70f7eac2"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.778724 8288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:42.779547 8303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.779759 8288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:42.779860 8288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
format_stamp: "Formatted at 2026-09-28 08:59:42 on dist-test-slave-286n"
server_key: "1d5e22f0ff8ff257dfecdf0d7eade977"
server_key_iv: "9adc70a8468776ca8b9f386c70f7eac2"
server_key_version: "encryptionkey@0"
I20260928 08:59:42.779944 8288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:42.792065 8288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:42.793993 8288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:42.794154 8288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:42.794631 8288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:42.794692 8288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:42.794739 8288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:42.794767 8288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:42.802652 8288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40431
I20260928 08:59:42.802809 8418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40431 every 8 connection(s)
I20260928 08:59:42.803117 8288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 08:59:42.805478 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.795157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36511 (local address 127.0.98.62:42051)
0928 08:59:42.795226 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.795228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.795413 (+ 185us) server_negotiation.cc:407] Connection header received
0928 08:59:42.795910 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.795912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.795926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.796005 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.796912 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.797869 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.799245 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.799480 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.799583 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.802247 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.802248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.802249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.802259 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.804030 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.804535 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.804536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.804537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.804594 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.804954 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.804954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.804955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.805078 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.805137 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.805347 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.805411 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:59:42.805687 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.795124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.3:36511)
0928 08:59:42.795284 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.795322 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.795808 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.796108 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.796113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.796488 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:42.796794 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.796799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.798020 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.798022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.799126 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.799129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.799614 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.799616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.799750 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.800378 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:42.800390 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:42.802117 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:42.804122 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.804128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.804129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.804451 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.804790 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.804791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.804791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.804892 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.805130 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:42.805132 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:42.805247 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.805322 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.805449 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260928 08:59:42.806321 8419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42051
I20260928 08:59:42.806420 8419 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:42.806592 8419 heartbeater.cc:511] Master 127.0.98.62:42051 requested a full tablet report, sending...
I20260928 08:59:42.807046 7935 ts_manager.cc:194] Registered new tserver with Master: a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:42.807488 7935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36511
I20260928 08:59:42.812026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8288
I20260928 08:59:42.812145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:42.812361 392 external_mini_cluster.cc:1428] Setting key 377408dad5a5d87df5c6f5275487c35d
I20260928 08:59:42.812726 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.807383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50445 (local address 127.0.98.62:42051)
0928 08:59:42.807432 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.807434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.807578 (+ 144us) server_negotiation.cc:407] Connection header received
0928 08:59:42.807625 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.807626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.807637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.807693 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.808000 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.808471 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.809241 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.809450 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.809533 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.810381 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.810383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.810383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.810390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.811540 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.811903 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.811904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.811904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.811964 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.812142 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.812143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.812143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.812281 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.812339 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.812540 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.812610 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 08:59:42.812736 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.807322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.98.3:50445)
0928 08:59:42.807486 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.807501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.807573 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.807694 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.807695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.807840 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:42.807915 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.807916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.808560 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.808561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.809159 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.809161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.809534 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.809535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.809601 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.810007 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:42.810011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:42.810295 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:42.811622 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.811623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.811623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.811798 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.812029 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:42.812030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:42.812031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:42.812089 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:42.812344 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:42.812345 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:42.812397 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.812510 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.812612 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260928 08:59:42.813330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:59:42.834560 8004 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:42.829159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.0.98.62:42051)
0928 08:59:42.829235 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.829237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.829247 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:42.829277 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.829278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.829289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.829364 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.829920 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.830408 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.831163 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.831350 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.831423 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.834468 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56500: BlockingRecv error: recv got EOF from 127.0.0.1:56500 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 08:59:42 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:42.841817 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.835324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.0.98.62:42051)
0928 08:59:42.835377 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.835379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.835468 (+ 89us) server_negotiation.cc:407] Connection header received
0928 08:59:42.835518 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.835518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.835526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.835567 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:42.835996 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.836436 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.837157 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.837325 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.837376 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.839385 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:42.839386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:42.839386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:42.839393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:42.840742 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.841105 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.841107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.841107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.841166 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:42.841410 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:42.841410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:42.841411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:42.841511 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:42.841582 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.841697 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.841754 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 08:59:42.843663 7936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:59:42.845582 7936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:42.851754 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.848380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.98.1:37007)
0928 08:59:42.848708 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.848713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.848722 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:42.848755 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.848756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.848770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.849012 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:42.849072 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.849713 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.850797 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.851340 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.851446 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.851526 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.851628 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.851665 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":82,"threads_started":1}
I20260928 08:59:42.851745 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.848239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37007 (local address 127.0.0.1:56724)
0928 08:59:42.848531 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.848541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.848583 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.848866 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.848867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.848870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:42.848933 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.848935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.849798 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.849799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.850700 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.850703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.851448 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.851449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.851526 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.851540 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.851612 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.851662 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":61,"threads_started":1}
I20260928 08:59:42.852727 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.848030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40431 (local address 127.0.0.1:58920)
0928 08:59:42.848206 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.848259 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.848328 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.848806 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.848807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.848809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:42.848852 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.848853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.850125 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.850125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.851407 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.851410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.852397 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.852398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.852465 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.852470 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.852520 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.852566 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":63,"thread_start_us":43,"threads_started":1}
I20260928 08:59:42.852738 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.848130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.0.98.2:35995)
0928 08:59:42.848344 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.848349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.848466 (+ 117us) server_negotiation.cc:407] Connection header received
0928 08:59:42.848550 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.848554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.848572 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.848670 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:42.848963 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.850020 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.851511 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.852239 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.852289 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.852361 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.852559 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.852603 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
I20260928 08:59:42.852770 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.848371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.98.3:40431)
0928 08:59:42.848542 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:42.848546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:42.848555 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:42.848598 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:42.848600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:42.848613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:42.848730 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:42.848928 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.850024 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.851513 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.852220 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.852292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.852361 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:42.852607 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 08:59:42.852664 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260928 08:59:42.852669 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 522c02f22754484e86e8a238eae18190 (DEFAULT_TABLE table=test-table [id=9fc214faf0bb46b390959d5ec8fb5447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:42.852945 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:42.848032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35995 (local address 127.0.0.1:57116)
0928 08:59:42.848393 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:42.848402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:42.848469 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:42.848704 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:42.848708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:42.848710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:42.848797 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.848800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.850125 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.850125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.851409 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:42.851410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:42.852416 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:42.852416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:42.852476 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:42.852479 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:42.852532 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 08:59:42.852565 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":65,"thread_start_us":98,"threads_started":1}
I20260928 08:59:42.853060 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522c02f22754484e86e8a238eae18190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:42.853415 8353 tablet_service.cc:1467] Processing CreateTablet for tablet 522c02f22754484e86e8a238eae18190 (DEFAULT_TABLE table=test-table [id=9fc214faf0bb46b390959d5ec8fb5447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:42.853492 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 522c02f22754484e86e8a238eae18190 (DEFAULT_TABLE table=test-table [id=9fc214faf0bb46b390959d5ec8fb5447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:42.853737 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522c02f22754484e86e8a238eae18190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:42.853741 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522c02f22754484e86e8a238eae18190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:42.856829 8447 tablet_bootstrap.cc:492] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Bootstrap starting.
I20260928 08:59:42.857396 8448 tablet_bootstrap.cc:492] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Bootstrap starting.
I20260928 08:59:42.857517 8447 tablet_bootstrap.cc:654] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:42.857725 8449 tablet_bootstrap.cc:492] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Bootstrap starting.
I20260928 08:59:42.857851 8447 log.cc:824] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:42.858085 8448 tablet_bootstrap.cc:654] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:42.858326 8448 log.cc:824] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:42.858387 8449 tablet_bootstrap.cc:654] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:42.858685 8447 tablet_bootstrap.cc:492] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: No bootstrap required, opened a new log
I20260928 08:59:42.858712 8449 log.cc:824] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:42.858767 8447 ts_tablet_manager.cc:1397] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:42.859104 8448 tablet_bootstrap.cc:492] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: No bootstrap required, opened a new log
I20260928 08:59:42.859200 8448 ts_tablet_manager.cc:1397] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:59:42.859437 8449 tablet_bootstrap.cc:492] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: No bootstrap required, opened a new log
I20260928 08:59:42.859513 8449 ts_tablet_manager.cc:1397] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:42.860908 8447 raft_consensus.cc:348] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:42.861084 8447 raft_consensus.cc:374] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:42.861122 8447 raft_consensus.cc:729] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Initialized, Role: FOLLOWER
I20260928 08:59:42.861002 8449 raft_consensus.cc:348] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:42.861161 8449 raft_consensus.cc:374] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:42.861193 8449 raft_consensus.cc:729] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Initialized, Role: FOLLOWER
I20260928 08:59:42.861238 8447 consensus_queue.cc:260] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:42.861326 8449 consensus_queue.cc:260] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:42.861501 8447 ts_tablet_manager.cc:1428] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:59:42.861408 8448 raft_consensus.cc:348] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:42.861544 8448 raft_consensus.cc:374] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:42.861583 8448 raft_consensus.cc:729] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed2aaa8e8f4d2f931cee8880aec313, State: Initialized, Role: FOLLOWER
I20260928 08:59:42.861626 8139 heartbeater.cc:503] Master 127.0.98.62:42051 was elected leader, sending a full tablet report...
I20260928 08:59:42.861708 8448 consensus_queue.cc:260] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:42.861792 8279 heartbeater.cc:503] Master 127.0.98.62:42051 was elected leader, sending a full tablet report...
I20260928 08:59:42.861969 8448 ts_tablet_manager.cc:1428] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:59:42.862006 8419 heartbeater.cc:503] Master 127.0.98.62:42051 was elected leader, sending a full tablet report...
I20260928 08:59:42.862317 8449 ts_tablet_manager.cc:1428] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260928 08:59:42.891825 8280 tablet.cc:2369] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:59:42.978335 8140 tablet.cc:2369] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:59:43.054525 8420 tablet.cc:2369] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:59:43.095275 8454 raft_consensus.cc:482] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:43.095414 8454 raft_consensus.cc:504] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:43.095858 8454 leader_election.cc:288] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431), e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007)
I20260928 08:59:43.099936 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.096115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33585 (local address 127.0.98.3:40431)
0928 08:59:43.096161 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.096163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.096178 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:59:43.096211 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.096212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.096224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.096325 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:43.096516 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.097292 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.098907 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.099477 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.099559 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.099570 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.099818 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.099864 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:59:43.099936 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.096132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45843 (local address 127.0.98.1:37007)
0928 08:59:43.096211 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.096214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.096284 (+ 70us) server_negotiation.cc:407] Connection header received
0928 08:59:43.096343 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.096344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.096357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.096413 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:43.096572 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.097324 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.098907 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.099475 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.099559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.099570 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.099818 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.099865 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:59:43.099891 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.096037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37007 (local address 127.0.98.2:45843)
0928 08:59:43.096200 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.096209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.096289 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.096432 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.096433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.096434 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:43.096498 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.096500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.097411 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.097412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.098767 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.098770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.099576 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.099577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.099670 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.099674 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.099736 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.099781 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":74,"thread_start_us":100,"threads_started":1}
I20260928 08:59:43.099896 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.096028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40431 (local address 127.0.98.2:33585)
0928 08:59:43.096094 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.096106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.096155 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.096318 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.096319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.096321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:43.096424 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.096441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.097411 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.097412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.098767 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.098770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.099576 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.099577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.099670 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.099673 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.099736 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.099781 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":104}
I20260928 08:59:43.100307 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522c02f22754484e86e8a238eae18190" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" is_pre_election: true
I20260928 08:59:43.100338 8093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522c02f22754484e86e8a238eae18190" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08bccd37dc4494081aca2849c570e0f" is_pre_election: true
I20260928 08:59:43.100495 8373 raft_consensus.cc:2454] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 0.
I20260928 08:59:43.100530 8093 raft_consensus.cc:2454] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 0.
I20260928 08:59:43.100775 8166 leader_election.cc:302] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, a4ed2aaa8e8f4d2f931cee8880aec313; no voters:
I20260928 08:59:43.100917 8454 raft_consensus.cc:2781] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:43.100973 8454 raft_consensus.cc:482] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:43.100998 8454 raft_consensus.cc:3037] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.101943 8454 raft_consensus.cc:504] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:43.102156 8454 leader_election.cc:288] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 election: Requested vote from peers a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431), e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007)
I20260928 08:59:43.102240 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522c02f22754484e86e8a238eae18190" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
I20260928 08:59:43.102236 8093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522c02f22754484e86e8a238eae18190" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08bccd37dc4494081aca2849c570e0f"
I20260928 08:59:43.102303 8373 raft_consensus.cc:3037] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.102324 8093 raft_consensus.cc:3037] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.102897 8373 raft_consensus.cc:2454] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 1.
I20260928 08:59:43.103049 8093 raft_consensus.cc:2454] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 1.
I20260928 08:59:43.103096 8166 leader_election.cc:302] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, a4ed2aaa8e8f4d2f931cee8880aec313; no voters:
I20260928 08:59:43.103193 8454 raft_consensus.cc:2781] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:43.103363 8454 raft_consensus.cc:686] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 LEADER]: Becoming Leader. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Running, Role: LEADER
I20260928 08:59:43.103470 8454 consensus_queue.cc:237] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } }
I20260928 08:59:43.104290 7934 catalog_manager.cc:5495] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:43.134405 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.131993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.0.98.2:35995)
0928 08:59:43.132065 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.132067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.132075 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:43.132109 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.132109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.132128 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.132189 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:43.132339 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.132868 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.133647 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.133831 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.133880 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.134071 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.134166 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.134226 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.134285 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.134303 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:59:43.136255 8373 raft_consensus.cc:1261] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Refusing update from remote peer 52f9f86c3ce84e7ba2dea6a83d3f399a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:43.136255 8093 raft_consensus.cc:1261] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Refusing update from remote peer 52f9f86c3ce84e7ba2dea6a83d3f399a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:43.136673 8454 consensus_queue.cc:1035] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:43.136821 8457 consensus_queue.cc:1035] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:43.138983 8460 mvcc.cc:205] Tried to move back new op lower bound from 7334240186923098112 to 7334240186792144896. Current Snapshot: MvccSnapshot[applied={T|T < 7334240186923098112}]
I20260928 08:59:43.139930 8461 mvcc.cc:205] Tried to move back new op lower bound from 7334240186923098112 to 7334240186792144896. Current Snapshot: MvccSnapshot[applied={T|T < 7334240186923098112 or (T in {7334240186923098112})}]
I20260928 08:59:43.140627 8462 mvcc.cc:205] Tried to move back new op lower bound from 7334240186923098112 to 7334240186792144896. Current Snapshot: MvccSnapshot[applied={T|T < 7334240186923098112 or (T in {7334240186923098112})}]
I20260928 08:59:43.141577 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:43.141664 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:43.142441 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Sending DeleteTablet for 3 replicas of tablet 522c02f22754484e86e8a238eae18190
I20260928 08:59:43.142753 8353 tablet_service.cc:1514] Processing DeleteTablet for tablet 522c02f22754484e86e8a238eae18190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58920
I20260928 08:59:43.142817 8073 tablet_service.cc:1514] Processing DeleteTablet for tablet 522c02f22754484e86e8a238eae18190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56724
I20260928 08:59:43.143013 8470 tablet_replica.cc:330] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: stopping tablet replica
I20260928 08:59:43.143023 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 522c02f22754484e86e8a238eae18190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57116
I20260928 08:59:43.142983 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:59:43.143110 8470 raft_consensus.cc:2229] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 08:59:43.143136 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:43.143218 8470 raft_consensus.cc:2258] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:43.143213 8469 tablet_replica.cc:330] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: stopping tablet replica
I20260928 08:59:43.143286 8469 raft_consensus.cc:2229] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:43.143357 8471 tablet_replica.cc:330] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: stopping tablet replica
I20260928 08:59:43.143424 8469 raft_consensus.cc:2258] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:43.143472 8471 raft_consensus.cc:2229] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:59:43.143918 8469 ts_tablet_manager.cc:1905] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:43.143963 8470 ts_tablet_manager.cc:1905] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:43.144201 8471 raft_consensus.cc:2258] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:43.144488 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 284df2c25f71468f8824f336fdd8597b (DEFAULT_TABLE table=test-table [id=2b34ee72a8c345f2999e81a85c582d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:43.144560 8353 tablet_service.cc:1467] Processing CreateTablet for tablet 284df2c25f71468f8824f336fdd8597b (DEFAULT_TABLE table=test-table [id=2b34ee72a8c345f2999e81a85c582d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:43.144610 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284df2c25f71468f8824f336fdd8597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.144673 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284df2c25f71468f8824f336fdd8597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.144886 8471 ts_tablet_manager.cc:1905] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:43.145916 8448 tablet_bootstrap.cc:492] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Bootstrap starting.
I20260928 08:59:43.146118 8447 tablet_bootstrap.cc:492] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Bootstrap starting.
I20260928 08:59:43.146298 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 284df2c25f71468f8824f336fdd8597b (DEFAULT_TABLE table=test-table [id=2b34ee72a8c345f2999e81a85c582d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:43.146396 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284df2c25f71468f8824f336fdd8597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.146412 8448 tablet_bootstrap.cc:654] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.146481 8447 tablet_bootstrap.cc:654] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.146814 8469 ts_tablet_manager.cc:1918] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:43.146874 8469 log.cc:1196] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/522c02f22754484e86e8a238eae18190
I20260928 08:59:43.146932 8471 ts_tablet_manager.cc:1918] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:43.146998 8471 log.cc:1196] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/522c02f22754484e86e8a238eae18190
I20260928 08:59:43.147156 8469 ts_tablet_manager.cc:1939] T 522c02f22754484e86e8a238eae18190 P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting consensus metadata
I20260928 08:59:43.147250 8447 tablet_bootstrap.cc:492] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: No bootstrap required, opened a new log
I20260928 08:59:43.147289 8447 ts_tablet_manager.cc:1397] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:43.147271 8471 ts_tablet_manager.cc:1939] T 522c02f22754484e86e8a238eae18190 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting consensus metadata
I20260928 08:59:43.147425 8447 raft_consensus.cc:348] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.147509 8447 raft_consensus.cc:374] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.147531 8447 raft_consensus.cc:729] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.147536 7922 catalog_manager.cc:4841] TS a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431): tablet 522c02f22754484e86e8a238eae18190 (table test-table [id=9fc214faf0bb46b390959d5ec8fb5447]) successfully deleted
I20260928 08:59:43.147574 8447 consensus_queue.cc:260] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.147675 8449 tablet_bootstrap.cc:492] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Bootstrap starting.
I20260928 08:59:43.147701 8447 ts_tablet_manager.cc:1428] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:43.147682 7921 catalog_manager.cc:4841] TS 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995): tablet 522c02f22754484e86e8a238eae18190 (table test-table [id=9fc214faf0bb46b390959d5ec8fb5447]) successfully deleted
I20260928 08:59:43.148015 8449 tablet_bootstrap.cc:654] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.148715 8449 tablet_bootstrap.cc:492] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: No bootstrap required, opened a new log
I20260928 08:59:43.148764 8449 ts_tablet_manager.cc:1397] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:43.148896 8449 raft_consensus.cc:348] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.148969 8449 raft_consensus.cc:374] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.148983 8449 raft_consensus.cc:729] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.149015 8449 consensus_queue.cc:260] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.149099 8449 ts_tablet_manager.cc:1428] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:43.149586 8470 ts_tablet_manager.cc:1918] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:43.149663 8470 log.cc:1196] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/522c02f22754484e86e8a238eae18190
I20260928 08:59:43.149926 8470 ts_tablet_manager.cc:1939] T 522c02f22754484e86e8a238eae18190 P e08bccd37dc4494081aca2849c570e0f: Deleting consensus metadata
I20260928 08:59:43.150328 7921 catalog_manager.cc:4841] TS e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007): tablet 522c02f22754484e86e8a238eae18190 (table test-table [id=9fc214faf0bb46b390959d5ec8fb5447]) successfully deleted
I20260928 08:59:43.155239 8448 tablet_bootstrap.cc:492] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: No bootstrap required, opened a new log
I20260928 08:59:43.155283 8448 ts_tablet_manager.cc:1397] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 08:59:43.155475 8448 raft_consensus.cc:348] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.155536 8448 raft_consensus.cc:374] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.155550 8448 raft_consensus.cc:729] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed2aaa8e8f4d2f931cee8880aec313, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.155603 8448 consensus_queue.cc:260] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.155696 8448 ts_tablet_manager.cc:1428] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:43.509938 8457 raft_consensus.cc:482] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:43.510056 8457 raft_consensus.cc:504] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.510272 8457 leader_election.cc:288] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:43.510500 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284df2c25f71468f8824f336fdd8597b" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" is_pre_election: true
I20260928 08:59:43.510583 8373 raft_consensus.cc:2454] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 0.
I20260928 08:59:43.510538 8093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284df2c25f71468f8824f336fdd8597b" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08bccd37dc4494081aca2849c570e0f" is_pre_election: true
I20260928 08:59:43.510697 8093 raft_consensus.cc:2454] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 0.
I20260928 08:59:43.510751 8166 leader_election.cc:302] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, a4ed2aaa8e8f4d2f931cee8880aec313; no voters:
I20260928 08:59:43.510838 8457 raft_consensus.cc:2781] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:43.510864 8457 raft_consensus.cc:482] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:43.510885 8457 raft_consensus.cc:3037] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.511659 8457 raft_consensus.cc:504] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.511871 8457 leader_election.cc:288] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 election: Requested vote from peers e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:43.511945 8093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284df2c25f71468f8824f336fdd8597b" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08bccd37dc4494081aca2849c570e0f"
I20260928 08:59:43.512033 8093 raft_consensus.cc:3037] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.512042 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284df2c25f71468f8824f336fdd8597b" candidate_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
I20260928 08:59:43.512115 8373 raft_consensus.cc:3037] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.512871 8373 raft_consensus.cc:2454] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 1.
I20260928 08:59:43.512871 8093 raft_consensus.cc:2454] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f9f86c3ce84e7ba2dea6a83d3f399a in term 1.
I20260928 08:59:43.513018 8166 leader_election.cc:302] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, a4ed2aaa8e8f4d2f931cee8880aec313; no voters:
I20260928 08:59:43.513151 8457 raft_consensus.cc:2781] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:43.513216 8457 raft_consensus.cc:686] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 LEADER]: Becoming Leader. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Running, Role: LEADER
I20260928 08:59:43.513298 8457 consensus_queue.cc:237] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.513981 7934 catalog_manager.cc:5495] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:59:43 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:43.608443 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.601465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42051 (local address 127.0.0.1:56522)
0928 08:59:43.601621 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.601641 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.601704 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.601904 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.601906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.602194 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:43.602240 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.602241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.602915 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.602915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.603431 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.603432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.603770 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.603771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.603813 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.604333 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:43.604338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:43.605776 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:43.607421 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:43.607428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:43.607430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:43.607636 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:43.607811 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:43.607812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:43.607812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:43.607870 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:43.608131 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:43.608137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:43.608181 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.608252 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.608350 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260928 08:59:43.608523 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.601513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.98.62:42051)
0928 08:59:43.601649 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.601651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.601702 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:59:43.601748 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.601749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.601759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.601832 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:43.602316 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.602821 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.603507 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.603693 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.603742 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.605850 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:43.605851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:43.605851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:43.605857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:43.607354 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:43.607702 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:43.607703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:43.607703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:43.607754 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:43.607956 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:43.607956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:43.607956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:43.608053 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:43.608104 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.608273 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.608397 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260928 08:59:43.610450 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56522:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:59:43.610672 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:43.612133 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 904ee55a2a3c49c4a38d1ffde62bbefa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=107701498d0d4b48b3d8b1335cbb017e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:59:43.612142 8353 tablet_service.cc:1467] Processing CreateTablet for tablet 904ee55a2a3c49c4a38d1ffde62bbefa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=107701498d0d4b48b3d8b1335cbb017e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:59:43.612165 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 904ee55a2a3c49c4a38d1ffde62bbefa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=107701498d0d4b48b3d8b1335cbb017e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:59:43.612289 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904ee55a2a3c49c4a38d1ffde62bbefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.612289 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904ee55a2a3c49c4a38d1ffde62bbefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.612308 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904ee55a2a3c49c4a38d1ffde62bbefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.613737 8449 tablet_bootstrap.cc:492] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: Bootstrap starting.
I20260928 08:59:43.613765 8447 tablet_bootstrap.cc:492] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: Bootstrap starting.
I20260928 08:59:43.614135 8447 tablet_bootstrap.cc:654] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.614135 8449 tablet_bootstrap.cc:654] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.614452 8448 tablet_bootstrap.cc:492] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: Bootstrap starting.
I20260928 08:59:43.614727 8449 tablet_bootstrap.cc:492] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: No bootstrap required, opened a new log
I20260928 08:59:43.614782 8449 ts_tablet_manager.cc:1397] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:43.614831 8448 tablet_bootstrap.cc:654] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.614893 8449 raft_consensus.cc:348] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.614977 8449 raft_consensus.cc:374] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.615015 8449 raft_consensus.cc:729] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.615065 8449 consensus_queue.cc:260] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.615175 8449 ts_tablet_manager.cc:1428] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:43.615219 8457 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.615314 8457 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.615432 8448 tablet_bootstrap.cc:492] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: No bootstrap required, opened a new log
I20260928 08:59:43.615473 8448 ts_tablet_manager.cc:1397] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:43.615614 8448 raft_consensus.cc:348] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.615671 8447 tablet_bootstrap.cc:492] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: No bootstrap required, opened a new log
I20260928 08:59:43.615682 8448 raft_consensus.cc:374] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.615710 8448 raft_consensus.cc:729] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed2aaa8e8f4d2f931cee8880aec313, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.615723 8447 ts_tablet_manager.cc:1397] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:43.615765 8448 consensus_queue.cc:260] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.615862 8448 ts_tablet_manager.cc:1428] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:43.615870 8447 raft_consensus.cc:348] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.615940 8447 raft_consensus.cc:374] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.615962 8447 raft_consensus.cc:729] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.615907 8455 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.616003 8455 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.616014 8447 consensus_queue.cc:260] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.616106 8447 ts_tablet_manager.cc:1428] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:43.616222 8453 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.616293 8453 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.790726 8453 raft_consensus.cc:482] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:43.790825 8453 raft_consensus.cc:504] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.791146 8453 leader_election.cc:288] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:43.794194 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.791417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44925 (local address 127.0.98.2:35995)
0928 08:59:43.791528 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.791530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.791582 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:59:43.791673 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.791674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.791682 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.791763 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:43.791950 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.792405 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.793479 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.793850 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.793907 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.793917 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.794077 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.794118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260928 08:59:43.794210 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.791369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35995 (local address 127.0.98.1:44925)
0928 08:59:43.791511 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.791522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.791610 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.791784 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.791785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.791787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:43.791870 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.791872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.792518 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.792519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.793384 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.793386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.793930 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.793931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.794000 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.794005 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.794063 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.794093 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260928 08:59:43.794302 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.791438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46895 (local address 127.0.98.3:40431)
0928 08:59:43.791534 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.791537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.791551 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:59:43.791641 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.791643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.791655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.791726 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:59:43.791893 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.792391 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.793487 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.793873 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.793965 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.793978 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.794172 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.794220 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260928 08:59:43.794531 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.791343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40431 (local address 127.0.98.1:46895)
0928 08:59:43.791495 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.791507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.791564 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.791738 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.791739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.791742 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:59:43.791819 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.791822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.792472 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.792473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.793383 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.793385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.793987 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.793988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.794090 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.794094 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.794149 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.794184 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":89,"thread_start_us":83,"threads_started":1}
I20260928 08:59:43.794586 8233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904ee55a2a3c49c4a38d1ffde62bbefa" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" is_pre_election: true
I20260928 08:59:43.794721 8233 raft_consensus.cc:2454] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 0.
I20260928 08:59:43.794803 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904ee55a2a3c49c4a38d1ffde62bbefa" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" is_pre_election: true
I20260928 08:59:43.794896 8373 raft_consensus.cc:2454] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 0.
I20260928 08:59:43.794926 8025 leader_election.cc:302] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, e08bccd37dc4494081aca2849c570e0f; no voters:
I20260928 08:59:43.795058 8453 raft_consensus.cc:2781] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:43.795105 8453 raft_consensus.cc:482] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:43.795132 8453 raft_consensus.cc:3037] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.795812 8453 raft_consensus.cc:504] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.795958 8453 leader_election.cc:288] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 election: Requested vote from peers 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:43.796096 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904ee55a2a3c49c4a38d1ffde62bbefa" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
I20260928 08:59:43.796191 8373 raft_consensus.cc:3037] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.796157 8233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904ee55a2a3c49c4a38d1ffde62bbefa" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a"
I20260928 08:59:43.796263 8233 raft_consensus.cc:3037] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:43.796813 8373 raft_consensus.cc:2454] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 1.
I20260928 08:59:43.797024 8026 leader_election.cc:302] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ed2aaa8e8f4d2f931cee8880aec313, e08bccd37dc4494081aca2849c570e0f; no voters:
I20260928 08:59:43.797107 8453 raft_consensus.cc:2781] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:43.797147 8233 raft_consensus.cc:2454] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 1.
I20260928 08:59:43.797209 8453 raft_consensus.cc:686] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [term 1 LEADER]: Becoming Leader. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Running, Role: LEADER
I20260928 08:59:43.797273 8453 consensus_queue.cc:237] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.797536 8490 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: TxnStatusTablet state changed. Reason: New leader e08bccd37dc4494081aca2849c570e0f. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.797624 8490 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:59:43.797777 8492 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:59:43.797900 8492 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:59:43.797849 7935 catalog_manager.cc:5495] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to e08bccd37dc4494081aca2849c570e0f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:43.844858 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.842012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.0.98.1:37007)
0928 08:59:43.842173 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.842175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.842181 (+ 6us) server_negotiation.cc:407] Connection header received
0928 08:59:43.842216 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.842217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.842240 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.842328 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:43.842487 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.843056 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.843903 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.844114 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.844200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.844387 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.844548 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.844632 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.844724 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.844746 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260928 08:59:43.844846 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.841874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37007 (local address 127.0.0.1:56726)
0928 08:59:43.841975 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.841989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.842053 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.842351 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.842352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.842354 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:59:43.842406 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.842409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.843149 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.843150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.843816 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.843819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.844201 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.844203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.844258 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.844271 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.844629 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.844659 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.844709 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.844768 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 08:59:43.846056 8373 raft_consensus.cc:1261] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Refusing update from remote peer e08bccd37dc4494081aca2849c570e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:43.846385 8233 raft_consensus.cc:1261] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Refusing update from remote peer e08bccd37dc4494081aca2849c570e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:43.846354 8453 consensus_queue.cc:1035] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:43.846637 8453 consensus_queue.cc:1035] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:43.846946 8455 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: TxnStatusTablet state changed. Reason: New leader e08bccd37dc4494081aca2849c570e0f. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.847050 8455 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.847676 8457 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: TxnStatusTablet state changed. Reason: New leader e08bccd37dc4494081aca2849c570e0f. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.847764 8457 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.848402 8490 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.848481 8490 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:59:43.848731 8457 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.848827 8457 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P 52f9f86c3ce84e7ba2dea6a83d3f399a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.849292 8490 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.849388 8490 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P e08bccd37dc4494081aca2849c570e0f: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:59:43.849764 8455 tablet_replica.cc:439] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } } }
I20260928 08:59:43.849848 8455 tablet_replica.cc:442] T 904ee55a2a3c49c4a38d1ffde62bbefa P a4ed2aaa8e8f4d2f931cee8880aec313: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:59:43.854663 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.852115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56435 (local address 127.0.98.1:37007)
0928 08:59:43.852153 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.852155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.852273 (+ 118us) server_negotiation.cc:407] Connection header received
0928 08:59:43.852356 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.852357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.852366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.852421 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:43.852519 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.853169 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.853963 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.854165 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.854240 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.854442 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.854474 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.854517 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.854580 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.854598 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:59:43.854698 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.852054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37007 (local address 127.0.98.2:56435)
0928 08:59:43.852196 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.852206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.852289 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.852424 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.852425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.852426 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:59:43.852472 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.852473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.853255 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.853256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.853850 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.853851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.854268 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.854268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.854346 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.854359 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.854527 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.854529 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.854576 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.854622 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 08:59:43.856870 8093 raft_consensus.cc:1261] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Refusing update from remote peer 52f9f86c3ce84e7ba2dea6a83d3f399a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:43.856815 8373 raft_consensus.cc:1261] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Refusing update from remote peer 52f9f86c3ce84e7ba2dea6a83d3f399a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:43.857074 8454 consensus_queue.cc:1035] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:43.857157 8457 consensus_queue.cc:1035] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:43.866096 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.862620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35017 (local address 127.0.98.2:35995)
0928 08:59:43.862654 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.862656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.862821 (+ 165us) server_negotiation.cc:407] Connection header received
0928 08:59:43.862991 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.862992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.863004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.863065 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:43.863143 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.863860 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.865280 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.865590 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.865664 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.865857 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.865891 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.865955 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.866002 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.866024 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 08:59:43.866119 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.862591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35995 (local address 127.0.98.1:35017)
0928 08:59:43.862742 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:43.862770 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:43.862847 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:43.863067 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:43.863068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:43.863069 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:59:43.863107 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.863107 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.863959 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.863960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.865206 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:43.865208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.865708 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.865709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:43.865762 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.865774 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:59:43.865962 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:59:43.865964 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 08:59:43.865998 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 08:59:43.866041 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260928 08:59:43.874071 7935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:43.874177 7935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:43.874786 7935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Sending DeleteTablet for 3 replicas of tablet 284df2c25f71468f8824f336fdd8597b
I20260928 08:59:43.875025 8073 tablet_service.cc:1514] Processing DeleteTablet for tablet 284df2c25f71468f8824f336fdd8597b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56724
I20260928 08:59:43.875126 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 284df2c25f71468f8824f336fdd8597b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57116
I20260928 08:59:43.877621 8353 tablet_service.cc:1514] Processing DeleteTablet for tablet 284df2c25f71468f8824f336fdd8597b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58920
I20260928 08:59:43.877807 8517 tablet_replica.cc:330] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: stopping tablet replica
I20260928 08:59:43.877883 8517 raft_consensus.cc:2229] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:43.877925 8517 raft_consensus.cc:2258] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:43.878171 8517 ts_tablet_manager.cc:1905] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:43.879757 8517 ts_tablet_manager.cc:1918] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:59:43.879827 8517 log.cc:1196] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/284df2c25f71468f8824f336fdd8597b
I20260928 08:59:43.880057 8517 ts_tablet_manager.cc:1939] T 284df2c25f71468f8824f336fdd8597b P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting consensus metadata
I20260928 08:59:43.880294 8515 tablet_replica.cc:330] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: stopping tablet replica
I20260928 08:59:43.880357 8515 raft_consensus.cc:2229] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:59:43.880365 7922 catalog_manager.cc:4841] TS a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431): tablet 284df2c25f71468f8824f336fdd8597b (table test-table [id=2b34ee72a8c345f2999e81a85c582d4d]) successfully deleted
I20260928 08:59:43.880419 8515 raft_consensus.cc:2258] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:43.880705 8515 ts_tablet_manager.cc:1905] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:43.882366 8515 ts_tablet_manager.cc:1918] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:59:43.882431 8515 log.cc:1196] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/284df2c25f71468f8824f336fdd8597b
I20260928 08:59:43.882665 8515 ts_tablet_manager.cc:1939] T 284df2c25f71468f8824f336fdd8597b P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting consensus metadata
I20260928 08:59:43.882839 8514 tablet_replica.cc:330] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: stopping tablet replica
I20260928 08:59:43.882915 8514 raft_consensus.cc:2229] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:43.882952 8514 raft_consensus.cc:2258] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:43.884837 8514 ts_tablet_manager.cc:1905] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:43.886216 8514 ts_tablet_manager.cc:1918] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:59:43.886449 8514 log.cc:1196] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/284df2c25f71468f8824f336fdd8597b
I20260928 08:59:43.886547 7921 catalog_manager.cc:4841] TS 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995): tablet 284df2c25f71468f8824f336fdd8597b (table test-table [id=2b34ee72a8c345f2999e81a85c582d4d]) successfully deleted
I20260928 08:59:43.886750 8514 ts_tablet_manager.cc:1939] T 284df2c25f71468f8824f336fdd8597b P e08bccd37dc4494081aca2849c570e0f: Deleting consensus metadata
I20260928 08:59:43.887089 7921 catalog_manager.cc:4841] TS e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007): tablet 284df2c25f71468f8824f336fdd8597b (table test-table [id=2b34ee72a8c345f2999e81a85c582d4d]) successfully deleted
Sep 28 08:59:43 dist-test-slave-286n krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:59:43 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:43 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:59:43.899175 8482 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:43.893800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.98.62:42051)
0928 08:59:43.893862 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.893864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.894021 (+ 157us) server_negotiation.cc:407] Connection header received
0928 08:59:43.894086 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.894087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.894102 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.894148 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:43.894739 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.895252 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.896047 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.896255 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.896340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.899089 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56536: BlockingRecv error: recv got EOF from 127.0.0.1:56536 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 08:59:43 dist-test-slave-286n krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 08:59:43.907310 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:43.899972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.0.98.62:42051)
0928 08:59:43.900014 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:43.900016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:43.900048 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:59:43.900114 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:43.900115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:43.900125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:43.900188 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:43.900611 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.901382 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.902065 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:43.902347 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:43.902420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:43.904482 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:43.904483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:43.904484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:43.904493 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:43.906158 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:43.906544 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:43.906545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:43.906546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:43.906609 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:43.906785 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:43.906786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:43.906787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:43.906927 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:43.907032 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:43.907127 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 08:59:43.907212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 08:59:43.908856 7935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:59:43.909045 7935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:43.910392 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 5106f0c553434f3eaee2aadfd9158f2d (DEFAULT_TABLE table=test-table [id=9d4e4e276b1544ac8f4fbc4597cecb0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:43.910406 8073 tablet_service.cc:1467] Processing CreateTablet for tablet 5106f0c553434f3eaee2aadfd9158f2d (DEFAULT_TABLE table=test-table [id=9d4e4e276b1544ac8f4fbc4597cecb0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:43.910406 8353 tablet_service.cc:1467] Processing CreateTablet for tablet 5106f0c553434f3eaee2aadfd9158f2d (DEFAULT_TABLE table=test-table [id=9d4e4e276b1544ac8f4fbc4597cecb0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:43.910531 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5106f0c553434f3eaee2aadfd9158f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.910531 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5106f0c553434f3eaee2aadfd9158f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.910531 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5106f0c553434f3eaee2aadfd9158f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:43.911834 8447 tablet_bootstrap.cc:492] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Bootstrap starting.
I20260928 08:59:43.911856 8448 tablet_bootstrap.cc:492] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Bootstrap starting.
I20260928 08:59:43.911953 8449 tablet_bootstrap.cc:492] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Bootstrap starting.
I20260928 08:59:43.912253 8449 tablet_bootstrap.cc:654] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.912277 8448 tablet_bootstrap.cc:654] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.912277 8447 tablet_bootstrap.cc:654] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:43.913127 8447 tablet_bootstrap.cc:492] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: No bootstrap required, opened a new log
I20260928 08:59:43.913172 8448 tablet_bootstrap.cc:492] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: No bootstrap required, opened a new log
I20260928 08:59:43.913188 8447 ts_tablet_manager.cc:1397] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:43.913223 8448 ts_tablet_manager.cc:1397] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:43.913254 8449 tablet_bootstrap.cc:492] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: No bootstrap required, opened a new log
I20260928 08:59:43.913300 8449 ts_tablet_manager.cc:1397] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:43.913347 8447 raft_consensus.cc:348] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.913363 8448 raft_consensus.cc:348] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.913408 8447 raft_consensus.cc:374] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.913425 8448 raft_consensus.cc:374] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.913391 8449 raft_consensus.cc:348] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.913429 8447 raft_consensus.cc:729] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.913448 8448 raft_consensus.cc:729] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed2aaa8e8f4d2f931cee8880aec313, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.913450 8449 raft_consensus.cc:374] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:43.913489 8449 raft_consensus.cc:729] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Initialized, Role: FOLLOWER
I20260928 08:59:43.913476 8447 consensus_queue.cc:260] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.913494 8448 consensus_queue.cc:260] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.913533 8449 consensus_queue.cc:260] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:43.913582 8448 ts_tablet_manager.cc:1428] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:43.913585 8447 ts_tablet_manager.cc:1428] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:43.913616 8449 ts_tablet_manager.cc:1428] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:44.209944 8507 raft_consensus.cc:482] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:44.210047 8507 raft_consensus.cc:504] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.210276 8507 leader_election.cc:288] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:44.210518 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5106f0c553434f3eaee2aadfd9158f2d" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" is_pre_election: true
I20260928 08:59:44.210532 8233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5106f0c553434f3eaee2aadfd9158f2d" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" is_pre_election: true
I20260928 08:59:44.210630 8373 raft_consensus.cc:2454] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 0.
I20260928 08:59:44.210650 8233 raft_consensus.cc:2454] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 0.
I20260928 08:59:44.210796 8025 leader_election.cc:302] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, e08bccd37dc4494081aca2849c570e0f; no voters:
I20260928 08:59:44.210909 8507 raft_consensus.cc:2781] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:44.210955 8507 raft_consensus.cc:482] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:44.210970 8507 raft_consensus.cc:3037] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:44.211700 8507 raft_consensus.cc:504] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.211829 8507 leader_election.cc:288] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 election: Requested vote from peers 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:44.211966 8233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5106f0c553434f3eaee2aadfd9158f2d" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a"
I20260928 08:59:44.212044 8233 raft_consensus.cc:3037] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:44.212013 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5106f0c553434f3eaee2aadfd9158f2d" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
I20260928 08:59:44.212113 8373 raft_consensus.cc:3037] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:44.212772 8233 raft_consensus.cc:2454] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 1.
I20260928 08:59:44.212797 8373 raft_consensus.cc:2454] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 1.
I20260928 08:59:44.212949 8025 leader_election.cc:302] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, e08bccd37dc4494081aca2849c570e0f; no voters:
I20260928 08:59:44.213075 8507 raft_consensus.cc:2781] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:44.213126 8507 raft_consensus.cc:686] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 1 LEADER]: Becoming Leader. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Running, Role: LEADER
I20260928 08:59:44.213171 8507 consensus_queue.cc:237] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.213781 7934 catalog_manager.cc:5495] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to e08bccd37dc4494081aca2849c570e0f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:44.267426 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:44.265223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.98.1:37007)
0928 08:59:44.265279 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:44.265281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:44.265290 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:44.265322 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:44.265323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:44.265334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:44.265382 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:44.265506 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:44.265978 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:44.266822 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:44.267009 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:44.267064 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:44.267214 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:44.267241 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:44.267279 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:44.267346 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:59:44.267364 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 08:59:44.268365 8233 raft_consensus.cc:1261] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Refusing update from remote peer e08bccd37dc4494081aca2849c570e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:44.268385 8373 raft_consensus.cc:1261] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Refusing update from remote peer e08bccd37dc4494081aca2849c570e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:44.268657 8453 consensus_queue.cc:1035] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:44.268785 8507 consensus_queue.cc:1035] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:44.272146 7935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:44.272249 7935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:44.272902 7935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Sending DeleteTablet for 3 replicas of tablet 5106f0c553434f3eaee2aadfd9158f2d
I20260928 08:59:44.273141 8073 tablet_service.cc:1514] Processing DeleteTablet for tablet 5106f0c553434f3eaee2aadfd9158f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56724
I20260928 08:59:44.273161 8353 tablet_service.cc:1514] Processing DeleteTablet for tablet 5106f0c553434f3eaee2aadfd9158f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58920
I20260928 08:59:44.273240 8514 tablet_replica.cc:330] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: stopping tablet replica
I20260928 08:59:44.273223 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 5106f0c553434f3eaee2aadfd9158f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57116
I20260928 08:59:44.273285 8514 raft_consensus.cc:2229] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:59:44.273339 8515 tablet_replica.cc:330] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: stopping tablet replica
I20260928 08:59:44.273368 8514 raft_consensus.cc:2258] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:44.273415 8515 raft_consensus.cc:2229] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:44.273456 8517 tablet_replica.cc:330] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: stopping tablet replica
I20260928 08:59:44.273473 8515 raft_consensus.cc:2258] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:44.273519 8517 raft_consensus.cc:2229] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:44.273588 8517 raft_consensus.cc:2258] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:44.274163 8514 ts_tablet_manager.cc:1905] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:44.274138 7935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:59:44.274299 7935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:59:44.276077 8517 ts_tablet_manager.cc:1905] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:44.276579 8073 tablet_service.cc:1467] Processing CreateTablet for tablet f2ceb2a449944726a13334b2b7f766e5 (DEFAULT_TABLE table=test-table [id=825ba7f1872e44d4b2cd7f3aea826339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:44.276660 8353 tablet_service.cc:1467] Processing CreateTablet for tablet f2ceb2a449944726a13334b2b7f766e5 (DEFAULT_TABLE table=test-table [id=825ba7f1872e44d4b2cd7f3aea826339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:44.276755 8353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ceb2a449944726a13334b2b7f766e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:44.276979 8212 tablet_service.cc:1467] Processing CreateTablet for tablet f2ceb2a449944726a13334b2b7f766e5 (DEFAULT_TABLE table=test-table [id=825ba7f1872e44d4b2cd7f3aea826339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:59:44.277083 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ceb2a449944726a13334b2b7f766e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:44.277752 8517 ts_tablet_manager.cc:1918] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:44.277808 8517 log.cc:1196] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/5106f0c553434f3eaee2aadfd9158f2d
I20260928 08:59:44.278045 8517 ts_tablet_manager.cc:1939] T 5106f0c553434f3eaee2aadfd9158f2d P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting consensus metadata
I20260928 08:59:44.278177 8448 tablet_bootstrap.cc:492] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Bootstrap starting.
I20260928 08:59:44.278240 8514 ts_tablet_manager.cc:1918] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:44.278290 8514 log.cc:1196] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/5106f0c553434f3eaee2aadfd9158f2d
I20260928 08:59:44.278522 8514 ts_tablet_manager.cc:1939] T 5106f0c553434f3eaee2aadfd9158f2d P e08bccd37dc4494081aca2849c570e0f: Deleting consensus metadata
I20260928 08:59:44.278661 8449 tablet_bootstrap.cc:492] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Bootstrap starting.
I20260928 08:59:44.278825 8448 tablet_bootstrap.cc:654] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:44.278999 8449 tablet_bootstrap.cc:654] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:44.279237 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ceb2a449944726a13334b2b7f766e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:44.279493 7922 catalog_manager.cc:4841] TS a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431): tablet 5106f0c553434f3eaee2aadfd9158f2d (table test-table [id=9d4e4e276b1544ac8f4fbc4597cecb0c]) successfully deleted
I20260928 08:59:44.279870 8448 tablet_bootstrap.cc:492] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: No bootstrap required, opened a new log
I20260928 08:59:44.279906 8448 ts_tablet_manager.cc:1397] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:59:44.280031 8448 raft_consensus.cc:348] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.280089 8448 raft_consensus.cc:374] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:44.280110 8448 raft_consensus.cc:729] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed2aaa8e8f4d2f931cee8880aec313, State: Initialized, Role: FOLLOWER
I20260928 08:59:44.280154 8448 consensus_queue.cc:260] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.280232 8448 ts_tablet_manager.cc:1428] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:44.280624 8447 tablet_bootstrap.cc:492] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f: Bootstrap starting.
I20260928 08:59:44.281006 8447 tablet_bootstrap.cc:654] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:44.281563 7921 catalog_manager.cc:4841] TS e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007): tablet 5106f0c553434f3eaee2aadfd9158f2d (table test-table [id=9d4e4e276b1544ac8f4fbc4597cecb0c]) successfully deleted
I20260928 08:59:44.281666 8447 tablet_bootstrap.cc:492] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f: No bootstrap required, opened a new log
I20260928 08:59:44.281721 8447 ts_tablet_manager.cc:1397] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:44.281836 8447 raft_consensus.cc:348] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.281911 8447 raft_consensus.cc:374] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:44.281935 8447 raft_consensus.cc:729] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Initialized, Role: FOLLOWER
I20260928 08:59:44.281965 8449 tablet_bootstrap.cc:492] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: No bootstrap required, opened a new log
I20260928 08:59:44.281978 8447 consensus_queue.cc:260] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.282013 8449 ts_tablet_manager.cc:1397] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 08:59:44.282065 8447 ts_tablet_manager.cc:1428] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:44.282126 8449 raft_consensus.cc:348] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.282217 8449 raft_consensus.cc:374] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:44.282243 8449 raft_consensus.cc:729] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f9f86c3ce84e7ba2dea6a83d3f399a, State: Initialized, Role: FOLLOWER
I20260928 08:59:44.282289 8449 consensus_queue.cc:260] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.282378 8449 ts_tablet_manager.cc:1428] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:44.283010 8515 ts_tablet_manager.cc:1905] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:44.284574 8515 ts_tablet_manager.cc:1918] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:59:44.284685 8515 log.cc:1196] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/5106f0c553434f3eaee2aadfd9158f2d
I20260928 08:59:44.284936 8515 ts_tablet_manager.cc:1939] T 5106f0c553434f3eaee2aadfd9158f2d P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting consensus metadata
I20260928 08:59:44.285288 7921 catalog_manager.cc:4841] TS 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995): tablet 5106f0c553434f3eaee2aadfd9158f2d (table test-table [id=9d4e4e276b1544ac8f4fbc4597cecb0c]) successfully deleted
I20260928 08:59:44.326483 8453 raft_consensus.cc:482] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:59:44.326589 8453 raft_consensus.cc:504] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.326737 8453 leader_election.cc:288] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:44.327003 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ceb2a449944726a13334b2b7f766e5" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" is_pre_election: true
I20260928 08:59:44.327097 8373 raft_consensus.cc:2454] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 0.
I20260928 08:59:44.327014 8233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ceb2a449944726a13334b2b7f766e5" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" is_pre_election: true
I20260928 08:59:44.327152 8233 raft_consensus.cc:2454] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 0.
I20260928 08:59:44.327255 8026 leader_election.cc:302] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ed2aaa8e8f4d2f931cee8880aec313, e08bccd37dc4494081aca2849c570e0f; no voters:
I20260928 08:59:44.327356 8453 raft_consensus.cc:2781] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:59:44.327411 8453 raft_consensus.cc:482] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:59:44.327446 8453 raft_consensus.cc:3037] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:44.328111 8453 raft_consensus.cc:504] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.328251 8453 leader_election.cc:288] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 election: Requested vote from peers 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995), a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431)
I20260928 08:59:44.328388 8233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ceb2a449944726a13334b2b7f766e5" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a"
I20260928 08:59:44.328403 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ceb2a449944726a13334b2b7f766e5" candidate_uuid: "e08bccd37dc4494081aca2849c570e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313"
I20260928 08:59:44.328490 8373 raft_consensus.cc:3037] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:44.328491 8233 raft_consensus.cc:3037] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:44.329142 8233 raft_consensus.cc:2454] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 1.
I20260928 08:59:44.329243 8373 raft_consensus.cc:2454] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08bccd37dc4494081aca2849c570e0f in term 1.
I20260928 08:59:44.329315 8025 leader_election.cc:302] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f9f86c3ce84e7ba2dea6a83d3f399a, e08bccd37dc4494081aca2849c570e0f; no voters:
I20260928 08:59:44.329461 8453 raft_consensus.cc:2781] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:44.329522 8453 raft_consensus.cc:686] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 1 LEADER]: Becoming Leader. State: Replica: e08bccd37dc4494081aca2849c570e0f, State: Running, Role: LEADER
I20260928 08:59:44.329568 8453 consensus_queue.cc:237] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } }
I20260928 08:59:44.330133 7934 catalog_manager.cc:5495] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to e08bccd37dc4494081aca2849c570e0f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e08bccd37dc4494081aca2849c570e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e08bccd37dc4494081aca2849c570e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 } health_report { overall_health: UNKNOWN } } }
I20260928 08:59:44.347254 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:44.344838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35067 (local address 127.0.98.1:37007)
0928 08:59:44.344877 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:44.344879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:44.344953 (+ 74us) server_negotiation.cc:407] Connection header received
0928 08:59:44.344973 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:44.344973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:44.344982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:44.345059 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:59:44.345182 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:44.345702 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:44.346572 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:44.346786 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:44.346874 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:44.347000 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:59:44.347032 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:59:44.347083 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:44.347139 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 08:59:44.347156 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":19}
I20260928 08:59:44.347254 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:44.344810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37007 (local address 127.0.98.1:35067)
0928 08:59:44.344860 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:44.344872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:44.344909 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:44.345060 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:44.345061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:44.345063 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:59:44.345118 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:44.345120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:44.345839 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:44.345840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:44.346467 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:44.346469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:44.346873 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:44.346874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:44.346926 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:44.346932 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:59:44.347081 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:59:44.347082 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 08:59:44.347126 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 08:59:44.347171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":2}
I20260928 08:59:44.349583 8233 raft_consensus.cc:1261] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Refusing update from remote peer e08bccd37dc4494081aca2849c570e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:44.349563 8373 raft_consensus.cc:1261] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Refusing update from remote peer e08bccd37dc4494081aca2849c570e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:59:44.349788 8507 consensus_queue.cc:1035] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f9f86c3ce84e7ba2dea6a83d3f399a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:44.349989 8490 consensus_queue.cc:1035] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ed2aaa8e8f4d2f931cee8880aec313" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:59:44.361224 7936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:44.361364 7936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:59:44.362063 7936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdb5c6d2949079f243b1df5434e49: Sending DeleteTablet for 3 replicas of tablet f2ceb2a449944726a13334b2b7f766e5
I20260928 08:59:44.362318 8353 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ceb2a449944726a13334b2b7f766e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58920
I20260928 08:59:44.362396 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ceb2a449944726a13334b2b7f766e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57116
I20260928 08:59:44.362432 8517 tablet_replica.cc:330] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: stopping tablet replica
I20260928 08:59:44.362483 8517 raft_consensus.cc:2229] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:44.362494 8515 tablet_replica.cc:330] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: stopping tablet replica
I20260928 08:59:44.362522 8517 raft_consensus.cc:2258] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:44.362560 8515 raft_consensus.cc:2229] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:59:44.362596 8515 raft_consensus.cc:2258] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:44.362665 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ceb2a449944726a13334b2b7f766e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:59:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56724
I20260928 08:59:44.362876 8514 tablet_replica.cc:330] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f: stopping tablet replica
I20260928 08:59:44.362926 8517 ts_tablet_manager.cc:1905] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:44.362943 8514 raft_consensus.cc:2229] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:59:44.362999 8514 raft_consensus.cc:2258] T f2ceb2a449944726a13334b2b7f766e5 P e08bccd37dc4494081aca2849c570e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:59:44.363073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8008
I20260928 08:59:44.363926 8515 ts_tablet_manager.cc:1905] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:59:44.364940 8517 ts_tablet_manager.cc:1918] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:59:44.365006 8517 log.cc:1196] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/f2ceb2a449944726a13334b2b7f766e5
I20260928 08:59:44.365238 8517 ts_tablet_manager.cc:1939] T f2ceb2a449944726a13334b2b7f766e5 P a4ed2aaa8e8f4d2f931cee8880aec313: Deleting consensus metadata
I20260928 08:59:44.365586 8515 ts_tablet_manager.cc:1918] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:59:44.365643 8515 log.cc:1196] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/f2ceb2a449944726a13334b2b7f766e5
I20260928 08:59:44.365850 8515 ts_tablet_manager.cc:1939] T f2ceb2a449944726a13334b2b7f766e5 P 52f9f86c3ce84e7ba2dea6a83d3f399a: Deleting consensus metadata
I20260928 08:59:44.365917 7922 catalog_manager.cc:4841] TS a4ed2aaa8e8f4d2f931cee8880aec313 (127.0.98.3:40431): tablet f2ceb2a449944726a13334b2b7f766e5 (table test-table [id=825ba7f1872e44d4b2cd7f3aea826339]) successfully deleted
I20260928 08:59:44.366281 7921 catalog_manager.cc:4841] TS 52f9f86c3ce84e7ba2dea6a83d3f399a (127.0.98.2:35995): tablet f2ceb2a449944726a13334b2b7f766e5 (table test-table [id=825ba7f1872e44d4b2cd7f3aea826339]) successfully deleted
W20260928 08:59:44.370431 7921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37007 (error 108)
I20260928 08:59:44.370692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8148
I20260928 08:59:44.371322 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:44.370784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37007 (local address 127.0.0.1:56746)
0928 08:59:44.371221 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:44.371272 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":37,"threads_started":1}
W20260928 08:59:44.371454 7921 catalog_manager.cc:4584] TS e08bccd37dc4494081aca2849c570e0f (127.0.98.1:37007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2ceb2a449944726a13334b2b7f766e5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37007: connect: Connection refused (error 111)
I20260928 08:59:44.377281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8288
I20260928 08:59:44.382121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 7904
2026-09-28T08:59:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3392 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:44 dist-test-slave-286n krb5kdc[8548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:44 dist-test-slave-286n krb5kdc[8548](info): set up 2 sockets
Sep 28 08:59:44 dist-test-slave-286n krb5kdc[8548](info): commencing operation
krb5kdc: starting...
W20260928 08:59:45.417181 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:45 dist-test-slave-286n krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:59:45.535717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33779
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:45.615588 8561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:45.615833 8561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:45.615870 8561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:45.617077 8561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:45.617146 8561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:45.617170 8561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:45.617204 8561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:45.617231 8561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:45.618700 8561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:45.618966 8561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:45.619189 8561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:45.624372 8569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:45.624445 8561 server_base.cc:1034] running on GCE node
W20260928 08:59:45.624392 8566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:45.624413 8567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:45.624832 8561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:45.625169 8561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:45.626340 8561 hybrid_clock.cc:648] HybridClock initialized: now 1790585985626323 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T08:59:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1250 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:59:45 dist-test-slave-286n krb5kdc[8574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:59:45 dist-test-slave-286n krb5kdc[8574](info): set up 2 sockets
Sep 28 08:59:45 dist-test-slave-286n krb5kdc[8574](info): commencing operation
krb5kdc: starting...
W20260928 08:59:46.701236 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 08:59:46 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:59:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 08:59:46.823061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:46.902868 8587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:46.903079 8587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:46.903102 8587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:46.904218 8587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:46.904253 8587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:46.904266 8587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:46.904279 8587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:46.904291 8587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:46.905820 8587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:46.906016 8587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:46.906184 8587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:46.911068 8593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:46.911051 8592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:46.911055 8595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:46.911283 8587 server_base.cc:1034] running on GCE node
I20260928 08:59:46.911521 8587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:46.911801 8587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:46.912979 8587 hybrid_clock.cc:648] HybridClock initialized: now 1790585986912972 us; error 38 us; skew 500 ppm
Sep 28 08:59:46 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:46.915465 8587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:59:46.915907 8587 webserver.cc:460] Webserver started at http://127.0.98.62:37435/ using document root <none> and password file <none>
I20260928 08:59:46.916136 8587 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:46.916204 8587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:46.916354 8587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:46.917526 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "385e622acc4949d5bfc354d0be65ebfa"
format_stamp: "Formatted at 2026-09-28 08:59:46 on dist-test-slave-286n"
server_key: "6367dc3606144aa599c764db3a55c84b"
server_key_iv: "814ceb069d78ea1d10b6d16a79535bd8"
server_key_version: "encryptionkey@0"
I20260928 08:59:46.917912 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "385e622acc4949d5bfc354d0be65ebfa"
format_stamp: "Formatted at 2026-09-28 08:59:46 on dist-test-slave-286n"
server_key: "6367dc3606144aa599c764db3a55c84b"
server_key_iv: "814ceb069d78ea1d10b6d16a79535bd8"
server_key_version: "encryptionkey@0"
I20260928 08:59:46.919270 8587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:59:46.920027 8602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:46.920261 8587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:59:46.920367 8587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "385e622acc4949d5bfc354d0be65ebfa"
format_stamp: "Formatted at 2026-09-28 08:59:46 on dist-test-slave-286n"
server_key: "6367dc3606144aa599c764db3a55c84b"
server_key_iv: "814ceb069d78ea1d10b6d16a79535bd8"
server_key_version: "encryptionkey@0"
I20260928 08:59:46.920439 8587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:46.947969 8587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:46.949932 8587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:46.950100 8587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:46.954406 8587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35473
I20260928 08:59:46.954445 8654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35473 every 8 connection(s)
I20260928 08:59:46.954980 8587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:46.955644 8655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:59:46.958101 8655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Bootstrap starting.
I20260928 08:59:46.958149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8587
I20260928 08:59:46.958235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 08:59:46.958437 392 external_mini_cluster.cc:1428] Setting key 494df61c2c3e608fb3ed4ef1107fe261
I20260928 08:59:46.959003 8655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Neither blocks nor log segments found. Creating new log.
I20260928 08:59:46.959416 8655 log.cc:824] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:46.960209 8655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: No bootstrap required, opened a new log
I20260928 08:59:46.962635 8655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:46.962792 8655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:59:46.962837 8655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385e622acc4949d5bfc354d0be65ebfa, State: Initialized, Role: FOLLOWER
I20260928 08:59:46.963006 8655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:46.963088 8655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:46.963116 8655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:46.963152 8655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:59:46.963800 8655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:46.963927 8655 leader_election.cc:302] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 385e622acc4949d5bfc354d0be65ebfa; no voters:
I20260928 08:59:46.964082 8655 leader_election.cc:288] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:59:46.964136 8660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:59:46.964272 8660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 LEADER]: Becoming Leader. State: Replica: 385e622acc4949d5bfc354d0be65ebfa, State: Running, Role: LEADER
I20260928 08:59:46.964423 8655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:46.964459 8660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:46.964916 8661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "385e622acc4949d5bfc354d0be65ebfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } } }
I20260928 08:59:46.964994 8661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:46.965210 8662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 385e622acc4949d5bfc354d0be65ebfa. Latest consensus state: current_term: 1 leader_uuid: "385e622acc4949d5bfc354d0be65ebfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } } }
I20260928 08:59:46.965565 8662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:46.965699 8666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:46.966200 8666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 08:59:46 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:46.967748 8666 catalog_manager.cc:1347] Generated new cluster ID: cd25b9b296234c30b73bb5b728f397ad
I20260928 08:59:46.967795 8666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:46.970582 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:46.959020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.98.62:35473)
0928 08:59:46.959201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:46.959211 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:46.959253 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:59:46.959808 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:46.959815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:46.959893 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:46.960066 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:46.960754 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:46.962151 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:46.963004 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:46.963601 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:46.963723 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:46.967184 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:46.967187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:46.967187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:46.967197 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:46.969066 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:46.969442 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:46.969445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:46.969445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:46.969532 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:46.969769 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:46.969770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:46.969770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:46.969998 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:46.970165 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:46.970279 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 08:59:46.970429 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 08:59:46.984071 8666 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:59:46.984616 8666 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:46.986966 8666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Generated new TSK 0
I20260928 08:59:46.987149 8666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 08:59:47.015478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35473
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:47.096300 8681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:47.096553 8681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:47.096593 8681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:47.097801 8681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:47.097873 8681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:47.097929 8681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:47.099470 8681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:47.099733 8681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:47.099972 8681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:47.104923 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.104923 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.104944 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.105252 8681 server_base.cc:1034] running on GCE node
I20260928 08:59:47.105480 8681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:47.105719 8681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:47.106860 8681 hybrid_clock.cc:648] HybridClock initialized: now 1790585987106841 us; error 41 us; skew 500 ppm
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:47.109501 8681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 08:59:47.109978 8681 webserver.cc:460] Webserver started at http://127.0.98.1:46303/ using document root <none> and password file <none>
I20260928 08:59:47.110219 8681 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:47.110289 8681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:47.110440 8681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:47.111486 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4ffcc6ea2a74ae4a8d5fcd70230b77d"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "7fcde288f632dfe0955ef35841188a39"
server_key_iv: "9c8d7214e1e8ad42e2994901ad122d50"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.111837 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4ffcc6ea2a74ae4a8d5fcd70230b77d"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "7fcde288f632dfe0955ef35841188a39"
server_key_iv: "9c8d7214e1e8ad42e2994901ad122d50"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.113205 8681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 08:59:47.113924 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.114109 8681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 08:59:47.114215 8681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "d4ffcc6ea2a74ae4a8d5fcd70230b77d"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "7fcde288f632dfe0955ef35841188a39"
server_key_iv: "9c8d7214e1e8ad42e2994901ad122d50"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.114289 8681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:47.127786 8681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:47.129737 8681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:47.129897 8681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:47.130119 8681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:47.130460 8681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:47.130517 8681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.130558 8681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:47.130586 8681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.136376 8681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42609
I20260928 08:59:47.136425 8809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42609 every 8 connection(s)
I20260928 08:59:47.136780 8681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:47.140558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8681
I20260928 08:59:47.140694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 08:59:47.140906 392 external_mini_cluster.cc:1428] Setting key 55e7c8a2dc18f5cabf74d9726b32a013
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:47.147274 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.137468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60739 (local address 127.0.98.62:35473)
0928 08:59:47.137533 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:47.137535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:47.137782 (+ 247us) server_negotiation.cc:407] Connection header received
0928 08:59:47.138843 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:47.138844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:47.138858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:47.138926 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:47.139583 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.140083 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.141345 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.141639 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.141712 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.144367 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:47.144369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:47.144370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:47.144376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:47.145814 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:47.146310 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:47.146311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:47.146311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:47.146356 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:47.146642 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:47.146643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:47.146643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:47.146740 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:47.146787 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:47.147140 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 08:59:47.147209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 08:59:47.147539 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.137466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:60739)
0928 08:59:47.137638 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:47.137672 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:47.138661 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:47.138984 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:47.138987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:47.139296 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:47.139501 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.139506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.140160 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.140161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.141277 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.141279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.141725 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.141727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.141877 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.142371 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:47.142381 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:47.144269 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:47.145897 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:47.145903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:47.145905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:47.146232 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:47.146423 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:47.146424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:47.146424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:47.146573 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:47.146813 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:47.146819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:47.146997 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 08:59:47.147124 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 08:59:47.147301 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260928 08:59:47.148113 8810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35473
I20260928 08:59:47.148206 8810 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:47.148407 8810 heartbeater.cc:511] Master 127.0.98.62:35473 requested a full tablet report, sending...
I20260928 08:59:47.148955 8616 ts_manager.cc:194] Registered new tserver with Master: d4ffcc6ea2a74ae4a8d5fcd70230b77d (127.0.98.1:42609)
I20260928 08:59:47.149556 8616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60739
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 08:59:47.167196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35473
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:47.247944 8815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:47.248198 8815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:47.248239 8815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:47.249526 8815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:47.249599 8815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:47.249644 8815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:47.251176 8815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:47.251471 8815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:47.251703 8815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:47.256812 8820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.256883 8823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.256918 8821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.257102 8815 server_base.cc:1034] running on GCE node
I20260928 08:59:47.257422 8815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:47.257656 8815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:47.258805 8815 hybrid_clock.cc:648] HybridClock initialized: now 1790585987258782 us; error 37 us; skew 500 ppm
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:47.261391 8815 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 08:59:47.261857 8815 webserver.cc:460] Webserver started at http://127.0.98.2:44595/ using document root <none> and password file <none>
I20260928 08:59:47.262037 8815 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:47.262079 8815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:47.262166 8815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:47.263155 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d6b0f7d22d146b99fafe488e3c889e9"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "d6562946c581eb0136f7503dd2dfe5ee"
server_key_iv: "518b8667d997b77d4431b633a83e0c7f"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.263506 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d6b0f7d22d146b99fafe488e3c889e9"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "d6562946c581eb0136f7503dd2dfe5ee"
server_key_iv: "518b8667d997b77d4431b633a83e0c7f"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.264776 8815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:59:47.265527 8830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.265822 8815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:47.265919 8815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "3d6b0f7d22d146b99fafe488e3c889e9"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "d6562946c581eb0136f7503dd2dfe5ee"
server_key_iv: "518b8667d997b77d4431b633a83e0c7f"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.266001 8815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:47.313575 8815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:47.315992 8815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:47.316164 8815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:47.316433 8815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:47.316792 8815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:47.316847 8815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.316890 8815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:47.316915 8815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.322815 8815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45931
I20260928 08:59:47.322862 8943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45931 every 8 connection(s)
I20260928 08:59:47.323210 8815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:47.332181 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.323968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59261 (local address 127.0.98.62:35473)
0928 08:59:47.324036 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:47.324039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:47.324205 (+ 166us) server_negotiation.cc:407] Connection header received
0928 08:59:47.325212 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:47.325212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:47.325221 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:47.325282 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:47.325847 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.326326 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.327259 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.327442 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.327493 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.329622 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:47.329623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:47.329623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:47.329629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:47.330968 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:47.331380 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:47.331380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:47.331381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:47.331435 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:47.331653 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:47.331654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:47.331654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:47.331750 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:47.331798 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:47.332057 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 08:59:47.332115 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:59:47.332440 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.323918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:59261)
0928 08:59:47.324075 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:47.324116 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:47.325119 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:47.325341 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:47.325344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:47.325587 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:47.325776 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.325780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.326402 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.326403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.327182 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.327184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.327513 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.327514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.327609 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.328137 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:47.328145 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:47.329543 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:47.331061 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:47.331067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:47.331069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:47.331306 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:47.331493 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:47.331494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:47.331494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:47.331598 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:47.331803 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:47.331807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:47.331948 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 08:59:47.332034 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:59:47.332155 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260928 08:59:47.332448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8815
I20260928 08:59:47.332553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 08:59:47.332788 392 external_mini_cluster.cc:1428] Setting key fc7c036cefabc12b1cdd7a17f8f5cfc4
I20260928 08:59:47.332970 8944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35473
I20260928 08:59:47.333093 8944 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:47.333278 8944 heartbeater.cc:511] Master 127.0.98.62:35473 requested a full tablet report, sending...
I20260928 08:59:47.333681 8616 ts_manager.cc:194] Registered new tserver with Master: 3d6b0f7d22d146b99fafe488e3c889e9 (127.0.98.2:45931)
I20260928 08:59:47.334069 8616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59261
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 08:59:47.371220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35473
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:47.452178 8949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:47.452379 8949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:47.452402 8949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:47.453701 8949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:47.453743 8949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:47.453775 8949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:47.455293 8949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:47.455538 8949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:47.455761 8949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:47.460610 8954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.460620 8957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.460635 8955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.460970 8949 server_base.cc:1034] running on GCE node
I20260928 08:59:47.461200 8949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:47.461439 8949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:47.462597 8949 hybrid_clock.cc:648] HybridClock initialized: now 1790585987462594 us; error 37 us; skew 500 ppm
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:47.470141 8949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 08:59:47.470657 8949 webserver.cc:460] Webserver started at http://127.0.98.3:42183/ using document root <none> and password file <none>
I20260928 08:59:47.470959 8949 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:47.471019 8949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:47.471153 8949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:59:47.472698 8949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "324ccdd972054d708d457a1b4f1db415"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "95010fdeda06433bf8aa2ac59f850291"
server_key_iv: "595c82433c985e2d9a55d47ffe809ad6"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.473109 8949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "324ccdd972054d708d457a1b4f1db415"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "95010fdeda06433bf8aa2ac59f850291"
server_key_iv: "595c82433c985e2d9a55d47ffe809ad6"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.478197 8949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260928 08:59:47.479336 8964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.479512 8949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:47.479598 8949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "324ccdd972054d708d457a1b4f1db415"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "95010fdeda06433bf8aa2ac59f850291"
server_key_iv: "595c82433c985e2d9a55d47ffe809ad6"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.479671 8949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:47.496924 8949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:47.499846 8949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:47.500008 8949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:47.500264 8949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:47.500626 8949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:47.500705 8949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.500748 8949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:47.500774 8949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.508569 8949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33487
I20260928 08:59:47.508692 9077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33487 every 8 connection(s)
I20260928 08:59:47.509064 8949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 08:59:47.516254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 8949
I20260928 08:59:47.516367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 08:59:47.516558 392 external_mini_cluster.cc:1428] Setting key bf2b25f4f02c6911d28000efb5af28bb
I20260928 08:59:47.519800 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.509855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40375 (local address 127.0.98.62:35473)
0928 08:59:47.509900 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:47.509902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:47.510167 (+ 265us) server_negotiation.cc:407] Connection header received
0928 08:59:47.511179 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:47.511181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:47.511193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:47.511250 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:47.511905 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.512434 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.513542 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.513764 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.513850 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.516185 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:47.516186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:47.516187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:47.516196 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:47.518249 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:47.518803 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:47.518804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:47.518805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:47.518854 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:47.519160 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:47.519160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:47.519160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:47.519261 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:47.519308 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:47.519672 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 08:59:47.519734 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 08:59:47.520140 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.509849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:40375)
0928 08:59:47.510025 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:47.510077 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:47.511065 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:47.511325 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:47.511329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:47.511618 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:47.511808 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.511812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.512530 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.512531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.513429 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.513431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.513858 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.513859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.513987 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.514557 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:47.514565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:47.516070 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:59:47.518336 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:47.518345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:47.518348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:47.518718 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:47.518925 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:59:47.518926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:59:47.518927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:59:47.519085 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:59:47.519328 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:59:47.519334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:59:47.519520 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 08:59:47.519658 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 08:59:47.519853 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260928 08:59:47.520706 9078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35473
I20260928 08:59:47.520792 9078 heartbeater.cc:464] Registering TS with master...
I20260928 08:59:47.521025 9078 heartbeater.cc:511] Master 127.0.98.62:35473 requested a full tablet report, sending...
I20260928 08:59:47.521476 8616 ts_manager.cc:194] Registered new tserver with Master: 324ccdd972054d708d457a1b4f1db415 (127.0.98.3:33487)
I20260928 08:59:47.522033 8616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40375
I20260928 08:59:47.528499 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 08:59:47.603461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8681
I20260928 08:59:47.608074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8815
I20260928 08:59:47.612013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8949
I20260928 08:59:47.616109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 8587
I20260928 08:59:47.620916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35473
--webserver_interface=127.0.98.62
--webserver_port=37435
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:47.700120 9087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:47.700315 9087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:47.700336 9087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:47.701452 9087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:59:47.701491 9087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:47.701505 9087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:47.701519 9087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:59:47.701529 9087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:59:47.702955 9087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:47.703137 9087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:47.703300 9087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:47.708006 9093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.708001 9095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.708034 9092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.708348 9087 server_base.cc:1034] running on GCE node
I20260928 08:59:47.708701 9087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:47.708992 9087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:47.710229 9087 hybrid_clock.cc:648] HybridClock initialized: now 1790585987710229 us; error 42 us; skew 500 ppm
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:47.712880 9087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 08:59:47.713372 9087 webserver.cc:460] Webserver started at http://127.0.98.62:37435/ using document root <none> and password file <none>
I20260928 08:59:47.713610 9087 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:47.713692 9087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:47.715047 9087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 08:59:47.715818 9102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.716017 9087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:47.716121 9087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "385e622acc4949d5bfc354d0be65ebfa"
format_stamp: "Formatted at 2026-09-28 08:59:46 on dist-test-slave-286n"
server_key: "6367dc3606144aa599c764db3a55c84b"
server_key_iv: "814ceb069d78ea1d10b6d16a79535bd8"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.716508 9087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:47.727116 9087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:47.729554 9087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:47.729722 9087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:47.733645 9087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35473
I20260928 08:59:47.733676 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35473 every 8 connection(s)
I20260928 08:59:47.734212 9087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 08:59:47.735210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9087
I20260928 08:59:47.735512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42609
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35473
--webserver_port=46303
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:59:47.735533 9155 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:59:47.736223 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Bootstrap starting.
I20260928 08:59:47.744918 9155 log.cc:824] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:59:47.746713 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:59:47.747007 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Bootstrap complete.
I20260928 08:59:47.749339 9155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:47.749564 9155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385e622acc4949d5bfc354d0be65ebfa, State: Initialized, Role: FOLLOWER
I20260928 08:59:47.749727 9155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:47.749796 9155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:59:47.749857 9155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:59:47.749908 9155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:59:47.750898 9155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:47.751012 9155 leader_election.cc:302] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 385e622acc4949d5bfc354d0be65ebfa; no voters:
I20260928 08:59:47.751195 9155 leader_election.cc:288] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:59:47.751226 9159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:59:47.751333 9159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [term 2 LEADER]: Becoming Leader. State: Replica: 385e622acc4949d5bfc354d0be65ebfa, State: Running, Role: LEADER
I20260928 08:59:47.751421 9159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } }
I20260928 08:59:47.751474 9155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:59:47.751667 9159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "385e622acc4949d5bfc354d0be65ebfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } } }
I20260928 08:59:47.751730 9159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:47.751839 9159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 385e622acc4949d5bfc354d0be65ebfa. Latest consensus state: current_term: 2 leader_uuid: "385e622acc4949d5bfc354d0be65ebfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385e622acc4949d5bfc354d0be65ebfa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35473 } } }
I20260928 08:59:47.751915 9159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa [sys.catalog]: This master's current role is: LEADER
I20260928 08:59:47.752250 9163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:59:47.752799 9163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:59:47.752934 9163 catalog_manager.cc:1259] Loaded cluster ID: cd25b9b296234c30b73bb5b728f397ad
I20260928 08:59:47.752974 9163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:59:47.754105 9163 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:59:47.754276 9163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 385e622acc4949d5bfc354d0be65ebfa: Loaded TSK: 0
I20260928 08:59:47.754480 9163 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:59:47.818413 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:47.818660 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:47.818701 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:47.820077 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:47.820137 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:47.820183 9157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 08:59:47.821817 9157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46303
--tserver_master_addrs=127.0.98.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:47.822101 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:47.822345 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:47.827796 9181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.827980 9182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.827829 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.828225 9157 server_base.cc:1034] running on GCE node
I20260928 08:59:47.828437 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:47.828715 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:47.829896 9157 hybrid_clock.cc:648] HybridClock initialized: now 1790585987829883 us; error 32 us; skew 500 ppm
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:47.832746 9157 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 08:59:47.833241 9157 webserver.cc:460] Webserver started at http://127.0.98.1:46303/ using document root <none> and password file <none>
I20260928 08:59:47.833491 9157 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:47.833576 9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:47.835004 9157 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:47.835654 9191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.835930 9157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:47.836040 9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "d4ffcc6ea2a74ae4a8d5fcd70230b77d"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "7fcde288f632dfe0955ef35841188a39"
server_key_iv: "9c8d7214e1e8ad42e2994901ad122d50"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.836510 9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:47.879935 9157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:47.882411 9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:47.882553 9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:47.882840 9157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:47.883167 9157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:47.883213 9157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.883244 9157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:47.883261 9157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.890280 9157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42609
I20260928 08:59:47.890326 9304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42609 every 8 connection(s)
I20260928 08:59:47.890674 9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 08:59:47.891012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9157
I20260928 08:59:47.891301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45931
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35473
--webserver_port=44595
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:47.899901 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.891535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:38669)
0928 08:59:47.891980 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:47.892028 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:47.892679 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:47.893232 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:47.893237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:47.894035 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:47.894223 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.894228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.895408 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.895408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.896192 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.896194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.896866 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.896867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.896994 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.897539 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:47.897546 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:47.899682 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":105,"threads_started":1}
W20260928 08:59:47.899940 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:47.891936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38669 (local address 127.0.98.62:35473)
0928 08:59:47.892176 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:47.892183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:47.892217 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:59:47.892945 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:47.892953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:47.893009 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:47.893156 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:47.894349 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.895284 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.896294 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.896759 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.896898 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.899797 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38669: BlockingRecv error: recv got EOF from 127.0.98.1:38669 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
W20260928 08:59:47.900192 9305 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35473: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:47.905056 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.900385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:55535)
0928 08:59:47.900664 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:47.900675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:47.900751 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:47.900897 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:47.900898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:47.901038 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:47.901081 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.901082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.901694 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.901695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.902203 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.902204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.902545 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.902546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.902613 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.903030 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:47.903035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:47.904983 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":86,"threads_started":1}
W20260928 08:59:47.905161 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:47.900548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55535 (local address 127.0.98.62:35473)
0928 08:59:47.900581 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:47.900583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:47.900738 (+ 155us) server_negotiation.cc:407] Connection header received
0928 08:59:47.900822 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:47.900823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:47.900836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:47.900891 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:47.901157 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.901614 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.902306 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.902478 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.902548 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.905067 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55535: BlockingRecv error: recv got EOF from 127.0.98.1:55535 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:47.909565 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:47.905313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:59617)
0928 08:59:47.905352 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:47.905362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:47.905402 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:47.905560 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:47.905560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:47.905692 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:47.905723 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.905724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.906271 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.906272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.907080 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:47.907081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.907425 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.907426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:47.907477 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.907831 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:47.907835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:47.909489 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 08:59:47.909618 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:47.905415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59617 (local address 127.0.98.62:35473)
0928 08:59:47.905445 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:47.905447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:47.905454 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:59:47.905474 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:47.905475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:47.905484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:47.905536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:47.905788 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.906205 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.907177 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:47.907348 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:47.907408 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:47.909563 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59617: BlockingRecv error: recv got EOF from 127.0.98.1:59617 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 08:59:47.909677 9305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:59:47.977408 9309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:47.977648 9309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:47.977687 9309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:47.978873 9309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:47.978956 9309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:47.979001 9309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 08:59:47.980448 9309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44595
--tserver_master_addrs=127.0.98.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:47.980756 9309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:47.981005 9309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:47.986114 9315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:47.986114 9318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.986397 9309 server_base.cc:1034] running on GCE node
W20260928 08:59:47.986129 9316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:47.986644 9309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:47.986838 9309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:47.987977 9309 hybrid_clock.cc:648] HybridClock initialized: now 1790585987987964 us; error 28 us; skew 500 ppm
Sep 28 08:59:47 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585987, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:47.990427 9309 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 08:59:47.991007 9309 webserver.cc:460] Webserver started at http://127.0.98.2:44595/ using document root <none> and password file <none>
I20260928 08:59:47.991253 9309 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:47.991333 9309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:47.992798 9309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:47.993355 9325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.993525 9309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:47.993620 9309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "3d6b0f7d22d146b99fafe488e3c889e9"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "d6562946c581eb0136f7503dd2dfe5ee"
server_key_iv: "518b8667d997b77d4431b633a83e0c7f"
server_key_version: "encryptionkey@0"
I20260928 08:59:47.993980 9309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:48.010007 9309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:48.012408 9309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:48.012573 9309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:48.012836 9309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:48.013149 9309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:48.013204 9309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:48.013247 9309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:48.013273 9309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:48.019155 9309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45931
I20260928 08:59:48.019196 9438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45931 every 8 connection(s)
I20260928 08:59:48.019537 9309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.027030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9309
W20260928 08:59:48.027223 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.020198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35919 (local address 127.0.98.62:35473)
0928 08:59:48.020243 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.020245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.020500 (+ 255us) server_negotiation.cc:407] Connection header received
0928 08:59:48.021071 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.021072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.021085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.021146 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.022090 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.022569 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.023567 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.023894 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.023987 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.027175 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35919: BlockingRecv error: recv got EOF from 127.0.98.2:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 08:59:48.027277 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.020186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:35919)
0928 08:59:48.020344 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.020387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.020973 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.021220 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.021224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.021817 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.022010 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.022015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.022645 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.022646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.023459 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.023460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.023992 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.023993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.024093 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.024712 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.024722 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.027059 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 08:59:48.027339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33487
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35473
--webserver_port=42183
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:59:48.027766 9439 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35473: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.032917 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.028067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:45735)
0928 08:59:48.028117 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.028126 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.028173 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.028311 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.028313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.028439 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.028484 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.028485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.029169 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.029171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.029950 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.029953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.030304 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.030305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.030378 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.030817 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.030823 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.032811 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 08:59:48.032951 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.028130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45735 (local address 127.0.98.62:35473)
0928 08:59:48.028190 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.028192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.028200 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:48.028226 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.028227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.028235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.028308 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.028567 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.029091 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.030052 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.030229 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.030288 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.032902 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45735: BlockingRecv error: recv got EOF from 127.0.98.2:45735 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.038316 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.033202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:40609)
0928 08:59:48.033382 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.033393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.033443 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.033669 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.033671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.033833 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.033874 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.033876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.034507 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.034508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.035410 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.035413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.035747 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.035748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.035815 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.036250 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.036257 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.038225 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
W20260928 08:59:48.038364 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.033252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40609 (local address 127.0.98.62:35473)
0928 08:59:48.033286 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.033288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.033451 (+ 163us) server_negotiation.cc:407] Connection header received
0928 08:59:48.033564 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.033565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.033575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.033652 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.033948 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.034425 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.035497 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.035672 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.035727 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.038300 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40609: BlockingRecv error: recv got EOF from 127.0.98.2:40609 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 08:59:48.038468 9439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:59:48.109205 9442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:59:48.109431 9442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:59:48.109472 9442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:59:48.110635 9442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:59:48.110700 9442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:59:48.110745 9442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 08:59:48.112188 9442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42183
--tserver_master_addrs=127.0.98.62:35473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 08:59:48.112452 9442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:59:48.112726 9442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:59:48.117843 9448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:59:48.117838 9449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:48.117858 9442 server_base.cc:1034] running on GCE node
W20260928 08:59:48.117815 9451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:59:48.118213 9442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:59:48.118479 9442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:59:48.119669 9442 hybrid_clock.cc:648] HybridClock initialized: now 1790585988119664 us; error 37 us; skew 500 ppm
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585988, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:59:48.122248 9442 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 08:59:48.122689 9442 webserver.cc:460] Webserver started at http://127.0.98.3:42183/ using document root <none> and password file <none>
I20260928 08:59:48.122929 9442 fs_manager.cc:359] Metadata directory not provided
I20260928 08:59:48.123013 9442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:59:48.124339 9442 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:59:48.124920 9458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:48.125087 9442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 08:59:48.125185 9442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "324ccdd972054d708d457a1b4f1db415"
format_stamp: "Formatted at 2026-09-28 08:59:47 on dist-test-slave-286n"
server_key: "95010fdeda06433bf8aa2ac59f850291"
server_key_iv: "595c82433c985e2d9a55d47ffe809ad6"
server_key_version: "encryptionkey@0"
I20260928 08:59:48.125528 9442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:59:48.154743 9442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:59:48.157177 9442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:59:48.157348 9442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:59:48.157595 9442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:59:48.157907 9442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:59:48.157969 9442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:48.158011 9442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:59:48.158038 9442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:59:48.163691 9442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33487
I20260928 08:59:48.163772 9571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33487 every 8 connection(s)
I20260928 08:59:48.164120 9442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.171003 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.164875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:37467)
0928 08:59:48.165059 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.165095 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.165559 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.165758 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.165762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.166312 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.166505 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.166509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.167125 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.167126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.167901 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.167903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.168238 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.168239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.168337 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.168856 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.168864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.170859 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
W20260928 08:59:48.171057 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.164901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37467 (local address 127.0.98.62:35473)
0928 08:59:48.164985 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.164988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.165180 (+ 192us) server_negotiation.cc:407] Connection header received
0928 08:59:48.165640 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.165641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.165651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.165700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.166578 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.167052 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.167975 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.168162 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.168214 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.171005 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37467: BlockingRecv error: recv got EOF from 127.0.98.3:37467 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 08:59:48.171366 9572 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35473 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35473: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260928 08:59:48.172814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9442
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.176625 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.171599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:35997)
0928 08:59:48.171649 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.171659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.171700 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.171924 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.171925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.172039 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.172086 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.172087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.172936 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.172936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.173649 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.173651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.174123 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.174124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.174170 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.174574 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.174580 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.176541 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 08:59:48.176723 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.171770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35997 (local address 127.0.98.62:35473)
0928 08:59:48.171820 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.171822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.171829 (+ 7us) server_negotiation.cc:407] Connection header received
0928 08:59:48.171850 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.171851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.171858 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.171908 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.172180 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.172851 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.173729 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.174046 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.174116 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.176613 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35997: BlockingRecv error: recv got EOF from 127.0.98.3:35997 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.181146 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.173259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.98.62:35473)
0928 08:59:48.173355 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.173357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.173472 (+ 115us) server_negotiation.cc:407] Connection header received
0928 08:59:48.173498 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.173499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.173516 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.173586 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.174302 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.174997 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.176089 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.176361 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.176413 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.177781 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:59:48.177786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:59:48.177786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:59:48.177801 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:59:48.179692 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:48.180051 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:48.180054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:48.180054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:48.180167 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:59:48.180382 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:59:48.180383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:59:48.180383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:59:48.180672 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:59:48.180799 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:59:48.180910 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 08:59:48.181034 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.181573 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.177198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:43951)
0928 08:59:48.177246 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.177257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.177299 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.177453 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.177454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.177569 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.177593 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.177594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.178282 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.178284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.179067 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.179069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.179388 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.179389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.179443 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.179747 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.179753 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.181532 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 08:59:48.181689 9308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.177264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43951 (local address 127.0.98.62:35473)
0928 08:59:48.177302 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.177304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.177312 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:48.177353 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.177354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.177363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.177416 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.177669 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.178199 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.179146 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.179316 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.179366 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.181621 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43951: BlockingRecv error: recv got EOF from 127.0.98.3:43951 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 08:59:48.181763 9572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:48 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:48.914791 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:48.910041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:34671)
0928 08:59:48.910197 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:48.910208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:48.910263 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:48.910394 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:48.910394 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:48.910548 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:48.910615 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.910616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.911313 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.911314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.911853 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:48.911855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.912215 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.912216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:48.912260 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.912786 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:48.912790 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:48.914689 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
W20260928 08:59:48.914832 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:48.910142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34671 (local address 127.0.98.62:35473)
0928 08:59:48.910273 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:48.910281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:48.910289 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:48.910312 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:48.910312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:48.910321 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:48.910381 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:48.910717 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.911232 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.911938 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:48.912148 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:48.912206 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:48.914759 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34671: BlockingRecv error: recv got EOF from 127.0.98.1:34671 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
Sep 28 08:59:49 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:49 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:49.043576 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:49.038843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:60651)
0928 08:59:49.038980 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:49.038990 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:49.039069 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:49.039172 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:49.039173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:49.039325 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:49.039371 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:49.039372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.039978 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.039978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:49.040723 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:49.040725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.041048 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.041049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:49.041086 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:49.041622 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:49.041626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:49.043469 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
W20260928 08:59:49.043623 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:49.038892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60651 (local address 127.0.98.62:35473)
0928 08:59:49.038966 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:49.038968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:49.039073 (+ 105us) server_negotiation.cc:407] Connection header received
0928 08:59:49.039110 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:49.039110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:49.039122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:49.039167 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:49.039429 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.039915 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.040794 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.040976 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.041030 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:49.043547 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60651: BlockingRecv error: recv got EOF from 127.0.98.2:60651 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 08:59:49 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:49 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:49.186646 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:49.182178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:56451)
0928 08:59:49.182336 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:49.182346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:49.182404 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:49.182559 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:49.182560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:49.182722 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:49.182771 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:49.182772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.183373 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.183374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:49.183983 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:49.183986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.184309 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.184310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:49.184348 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:49.184855 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:49.184859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:49.186579 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
W20260928 08:59:49.186698 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:49.182255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56451 (local address 127.0.98.62:35473)
0928 08:59:49.182302 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:49.182304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:49.182408 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:59:49.182462 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:49.182462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:49.182475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:49.182519 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:49.182833 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.183308 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.184063 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.184246 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.184297 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:49.186652 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56451: BlockingRecv error: recv got EOF from 127.0.98.3:56451 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 08:59:49 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:49 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:49.919991 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:49.915313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:34071)
0928 08:59:49.915447 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:49.915457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:49.915505 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:49.915750 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:49.915751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:49.915957 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:49.916018 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:49.916025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.916820 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.916821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:49.917330 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:49.917331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.917727 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.917728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:49.917767 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:49.918238 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:49.918243 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:49.919933 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
W20260928 08:59:49.920161 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:49.915397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34071 (local address 127.0.98.62:35473)
0928 08:59:49.915604 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:49.915606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:49.915615 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:49.915637 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:49.915637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:49.915648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:49.915721 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:49.916095 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.916749 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.917399 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:49.917651 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:49.917712 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:49.920043 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34071: BlockingRecv error: recv got EOF from 127.0.98.1:34071 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
Sep 28 08:59:50 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:50 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:50.048972 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:50.044149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:41017)
0928 08:59:50.044304 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:50.044326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:50.044433 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:50.044651 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:50.044652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:50.044855 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:50.044923 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:50.044926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.045811 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.045812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:50.046329 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:50.046330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.046683 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.046684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:50.046723 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:50.047185 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:50.047189 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:50.048917 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
W20260928 08:59:50.049119 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:50.044256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41017 (local address 127.0.98.62:35473)
0928 08:59:50.044309 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:50.044311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:50.044406 (+ 95us) server_negotiation.cc:407] Connection header received
0928 08:59:50.044505 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:50.044506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:50.044521 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:50.044625 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:50.045055 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.045747 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.046405 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.046591 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.046684 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:50.049037 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41017: BlockingRecv error: recv got EOF from 127.0.98.2:41017 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 08:59:50 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:50 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:50.191881 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:50.187190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:41547)
0928 08:59:50.187327 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:50.187338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:50.187394 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:50.187550 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:50.187551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:50.187733 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:50.187778 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:50.187780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.188457 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.188459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:50.189070 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:50.189073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.189482 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.189483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:50.189543 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:50.190024 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:50.190028 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:50.191821 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260928 08:59:50.192027 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:50.187338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41547 (local address 127.0.98.62:35473)
0928 08:59:50.187383 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:50.187385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:50.187393 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:50.187446 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:50.187447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:50.187460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:50.187543 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:50.187864 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.188363 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.189191 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.189378 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.189470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:50.191939 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41547: BlockingRecv error: recv got EOF from 127.0.98.3:41547 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 08:59:50 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:50 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:50.924861 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:50.920446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:33267)
0928 08:59:50.920607 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:50.920625 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:50.920724 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:50.920880 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:50.920881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:50.921037 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:50.921084 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:50.921085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.921710 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.921710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:50.922228 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:50.922229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.922544 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.922545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:50.922581 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:50.923086 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:50.923090 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:50.924796 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
W20260928 08:59:50.925030 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:50.920509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33267 (local address 127.0.98.62:35473)
0928 08:59:50.920680 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:50.920682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:50.920708 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:59:50.920787 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:50.920788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:50.920799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:50.920862 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:50.921142 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.921643 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.922295 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:50.922481 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:50.922532 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:50.924942 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33267: BlockingRecv error: recv got EOF from 127.0.98.1:33267 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
Sep 28 08:59:51 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:51 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:51.054442 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:51.049396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:48575)
0928 08:59:51.049565 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:51.049577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:51.049662 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:51.049831 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:51.049831 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:51.050010 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:51.050056 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:51.050057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.050716 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.050717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:51.051363 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:51.051365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.051737 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.051738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:51.051795 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:51.052396 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:51.052401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:51.054376 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
W20260928 08:59:51.054513 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:51.049485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48575 (local address 127.0.98.62:35473)
0928 08:59:51.049557 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:51.049560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:51.049642 (+ 82us) server_negotiation.cc:407] Connection header received
0928 08:59:51.049725 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:51.049727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:51.049740 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:51.049816 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:51.050131 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.050632 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.051452 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.051650 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.051736 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:51.054467 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48575: BlockingRecv error: recv got EOF from 127.0.98.2:48575 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 08:59:51 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:51 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:51.197089 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:51.192339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:36877)
0928 08:59:51.192518 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:51.192529 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:51.192617 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:51.192815 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:51.192816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:51.192989 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:51.193035 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:51.193037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.193712 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.193713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:51.194247 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:51.194248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.194619 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.194620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:51.194676 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:51.195205 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:51.195216 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:51.197001 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
W20260928 08:59:51.197162 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:51.192425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36877 (local address 127.0.98.62:35473)
0928 08:59:51.192501 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:51.192503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:51.192597 (+ 94us) server_negotiation.cc:407] Connection header received
0928 08:59:51.192714 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:51.192715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:51.192729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:51.192795 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:51.193121 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.193621 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.194339 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.194550 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.194622 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:51.197113 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36877: BlockingRecv error: recv got EOF from 127.0.98.3:36877 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 08:59:51 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:51 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:51.930016 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:51.925352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:32855)
0928 08:59:51.925520 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:51.925533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:51.925625 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:51.925838 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:51.925839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:51.926032 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:51.926081 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:51.926083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.926718 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.926719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:51.927244 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:51.927245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.927605 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.927606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:51.927661 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:51.928103 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:51.928107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:51.929959 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
W20260928 08:59:51.930147 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:51.925429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32855 (local address 127.0.98.62:35473)
0928 08:59:51.925536 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:51.925538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:51.925606 (+ 68us) server_negotiation.cc:407] Connection header received
0928 08:59:51.925704 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:51.925705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:51.925718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:51.925815 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:51.926146 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.926647 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.927325 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:51.927532 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:51.927584 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:51.930059 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32855: BlockingRecv error: recv got EOF from 127.0.98.1:32855 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
Sep 28 08:59:52 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:52 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:52.059850 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:52.055073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:55461)
0928 08:59:52.055239 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:52.055250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:52.055349 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:52.055579 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:52.055580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:52.055780 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:52.055828 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:52.055830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.056455 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.056456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:52.057046 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:52.057048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.057359 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.057360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:52.057401 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:52.057899 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:52.057911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:52.059764 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
W20260928 08:59:52.059958 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:52.055156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55461 (local address 127.0.98.62:35473)
0928 08:59:52.055217 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:52.055220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:52.055338 (+ 118us) server_negotiation.cc:407] Connection header received
0928 08:59:52.055438 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:52.055439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:52.055452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:52.055531 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:52.055896 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.056387 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.057112 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.057295 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.057350 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:52.059894 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55461: BlockingRecv error: recv got EOF from 127.0.98.2:55461 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 08:59:52 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:52 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:52.203895 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:52.197608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:48571)
0928 08:59:52.197775 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:52.197787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:52.197855 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:52.198069 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:52.198070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:52.198302 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:52.198379 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:52.198382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.199353 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.199355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:52.200258 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:52.200262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.200789 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.200791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:52.200860 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:52.201474 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:52.201479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:52.203793 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
W20260928 08:59:52.203955 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:52.197658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48571 (local address 127.0.98.62:35473)
0928 08:59:52.197702 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:52.197704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:52.197858 (+ 154us) server_negotiation.cc:407] Connection header received
0928 08:59:52.197951 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:52.197952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:52.197966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:52.198035 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:52.198479 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.199263 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.200379 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.200695 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.200765 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:52.203886 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48571: BlockingRecv error: recv got EOF from 127.0.98.3:48571 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 08:59:52 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:52 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:52.940373 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:52.930490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:52285)
0928 08:59:52.930682 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:52.930694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:52.930751 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:52.930904 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:52.930905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:52.931075 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:52.931125 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:52.931126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.931790 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.931791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:52.932317 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:52.932318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.932662 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.932663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:52.932704 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:52.938541 (+ 5837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:52.938546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:52.940315 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":116,"threads_started":1}
W20260928 08:59:52.940519 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:52.930589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52285 (local address 127.0.98.62:35473)
0928 08:59:52.930721 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:52.930723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:52.930738 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:59:52.930812 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:52.930813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:52.930823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:52.930892 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:52.931210 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.931714 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.932387 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:52.932579 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:52.932629 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:52.940426 (+ 7797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52285: BlockingRecv error: recv got EOF from 127.0.98.1:52285 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
Sep 28 08:59:53 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:53 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:53.065001 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:53.060325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:55949)
0928 08:59:53.060449 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:53.060463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:53.060519 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:53.060716 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:53.060717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:53.060893 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:53.060940 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:53.060941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.061648 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.061649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:53.062206 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:53.062207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.062576 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.062577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:53.062635 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:53.063128 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:53.063133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:53.064930 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
W20260928 08:59:53.065070 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:53.060432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55949 (local address 127.0.98.62:35473)
0928 08:59:53.060506 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:53.060509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:53.060518 (+ 9us) server_negotiation.cc:407] Connection header received
0928 08:59:53.060587 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:53.060588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:53.060600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:53.060706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:53.061020 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.061542 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.062290 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.062489 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.062583 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:53.065023 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55949: BlockingRecv error: recv got EOF from 127.0.98.2:55949 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 08:59:53 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:53 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:53.209618 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:53.204743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:56517)
0928 08:59:53.204989 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:53.204999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:53.205083 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:53.205245 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:53.205246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:53.205427 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:53.205473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:53.205474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.206150 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.206152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:53.206693 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:53.206695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.207077 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.207077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:53.207142 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:53.207698 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:53.207702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:53.209520 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
W20260928 08:59:53.209790 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:53.204845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56517 (local address 127.0.98.62:35473)
0928 08:59:53.204922 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:53.204925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:53.205066 (+ 141us) server_negotiation.cc:407] Connection header received
0928 08:59:53.205171 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:53.205172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:53.205186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:53.205229 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:53.205564 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.206065 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.206780 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.206990 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.207076 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:53.209644 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56517: BlockingRecv error: recv got EOF from 127.0.98.3:56517 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 08:59:53 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:53 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:53.945282 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:53.940813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:49393)
0928 08:59:53.941016 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:53.941029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:53.941103 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:53.941235 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:53.941236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:53.941397 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:53.941443 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:53.941444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.942196 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.942197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:53.942751 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:53.942753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.943082 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.943082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:53.943125 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:53.943554 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:53.943558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:53.945215 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
W20260928 08:59:53.945384 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:53.940869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49393 (local address 127.0.98.62:35473)
0928 08:59:53.941045 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:53.941047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:53.941079 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:59:53.941154 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:53.941155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:53.941164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:53.941223 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:53.941502 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.942112 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.942824 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:53.943014 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:53.943066 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:53.945307 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49393: BlockingRecv error: recv got EOF from 127.0.98.1:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":55,"threads_started":1}
Sep 28 08:59:54 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:54 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:54.070779 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:54.065549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:56853)
0928 08:59:54.065732 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:54.065746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:54.065848 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:54.066095 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:54.066096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:54.066277 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:54.066326 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:54.066329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.067029 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.067031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:54.067578 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:54.067580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.067972 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.067973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:54.068047 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:54.068517 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:54.068522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:54.070677 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
W20260928 08:59:54.070845 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:54.065665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56853 (local address 127.0.98.62:35473)
0928 08:59:54.065738 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:54.065741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:54.065828 (+ 87us) server_negotiation.cc:407] Connection header received
0928 08:59:54.065945 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:54.065946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:54.065967 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:54.066024 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:54.066418 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.066932 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.067682 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.067881 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.067971 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:54.070779 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56853: BlockingRecv error: recv got EOF from 127.0.98.2:56853 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 08:59:54 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:54 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:54.214959 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:54.210151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:35995)
0928 08:59:54.210350 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:54.210362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:54.210436 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:54.210590 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:54.210591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:54.210776 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:54.210822 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:54.210823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.211512 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.211513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:54.212160 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:54.212162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.212527 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.212528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:54.212584 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:54.213090 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:54.213094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:54.214902 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
W20260928 08:59:54.215044 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:54.210247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35995 (local address 127.0.98.62:35473)
0928 08:59:54.210340 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:54.210343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:54.210416 (+ 73us) server_negotiation.cc:407] Connection header received
0928 08:59:54.210500 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:54.210501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:54.210515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:54.210583 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:54.210911 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.211427 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.212255 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.212443 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.212525 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:54.214995 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35995: BlockingRecv error: recv got EOF from 127.0.98.3:35995 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:59:54 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:54 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:54.950042 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:54.945725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:42503)
0928 08:59:54.945881 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:54.945892 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:54.945947 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:54.946080 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:54.946081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:54.946245 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:54.946291 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:54.946292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.946923 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.946924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:54.947498 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:54.947500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.947811 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.947812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:54.947852 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:54.948260 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:54.948264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:54.949982 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
W20260928 08:59:54.950147 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:54.945838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42503 (local address 127.0.98.62:35473)
0928 08:59:54.945976 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:54.945978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:54.945986 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:54.946006 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:54.946006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:54.946016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:54.946066 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:54.946360 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.946857 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.947570 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:54.947749 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:54.947808 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:54.950068 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42503: BlockingRecv error: recv got EOF from 127.0.98.1:42503 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
Sep 28 08:59:55 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:55 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:55.075605 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:55.071220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:56979)
0928 08:59:55.071379 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:55.071390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:55.071451 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:55.071570 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:55.071570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:55.071725 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:55.071774 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:55.071775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.072374 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.072376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:55.072916 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:55.072917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.073232 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.073233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:55.073270 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:55.073782 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:55.073788 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:55.075547 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
W20260928 08:59:55.075703 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:55.071288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56979 (local address 127.0.98.62:35473)
0928 08:59:55.071367 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:55.071369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:55.071453 (+ 84us) server_negotiation.cc:407] Connection header received
0928 08:59:55.071495 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:55.071495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:55.071506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:55.071557 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:55.071831 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.072311 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.072983 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.073168 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.073228 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:55.075636 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56979: BlockingRecv error: recv got EOF from 127.0.98.2:56979 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 08:59:55 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:55 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:55.220013 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:55.215547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:45625)
0928 08:59:55.215730 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:55.215740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:55.215794 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:55.215919 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:55.215920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:55.216076 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:55.216124 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:55.216125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.216774 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.216775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:55.217286 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:55.217288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.217597 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.217598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:55.217636 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:55.218142 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:55.218146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:55.219919 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
W20260928 08:59:55.220050 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:55.215634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45625 (local address 127.0.98.62:35473)
0928 08:59:55.215704 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:55.215706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:55.215798 (+ 92us) server_negotiation.cc:407] Connection header received
0928 08:59:55.215854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:55.215855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:55.215868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:55.215907 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:55.216182 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.216709 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.217357 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.217534 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.217587 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:55.219990 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45625: BlockingRecv error: recv got EOF from 127.0.98.3:45625 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 08:59:55 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:55 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:55.955529 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:55.950533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:38001)
0928 08:59:55.950665 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:55.950676 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:55.950743 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:55.950938 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:55.950939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:55.951112 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:55.951159 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:55.951160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.951884 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.951885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:55.952466 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:55.952468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.952899 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.952900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:55.952966 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:55.953628 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:55.953633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:55.955470 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20260928 08:59:55.955644 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:55.950705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38001 (local address 127.0.98.62:35473)
0928 08:59:55.950818 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:55.950820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:55.950828 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:55.950849 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:55.950849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:55.950858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:55.950923 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:55.951225 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.951806 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.952538 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:55.952820 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:55.952873 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:55.955564 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38001: BlockingRecv error: recv got EOF from 127.0.98.1:38001 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
Sep 28 08:59:56 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:56 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:56.080770 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:56.076114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:44693)
0928 08:59:56.076250 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:56.076263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:56.076328 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:56.076471 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:56.076471 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:56.076660 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:56.076707 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:56.076709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.077451 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.077452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:56.077978 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:56.077979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.078295 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.078296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:56.078337 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:56.078889 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:56.078894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:56.080704 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260928 08:59:56.080811 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:56.076194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44693 (local address 127.0.98.62:35473)
0928 08:59:56.076263 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:56.076265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:56.076337 (+ 72us) server_negotiation.cc:407] Connection header received
0928 08:59:56.076374 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:56.076375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:56.076386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:56.076452 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:56.076757 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.077376 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.078053 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.078231 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.078281 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:56.080763 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44693: BlockingRecv error: recv got EOF from 127.0.98.2:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 08:59:56 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:56 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:56.225390 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:56.220544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:36831)
0928 08:59:56.220723 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:56.220737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:56.220835 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:56.220963 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:56.220964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:56.221141 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:56.221188 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:56.221190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.221884 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.221885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:56.222462 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:56.222465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.222909 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.222910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:56.222971 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:56.223510 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:56.223514 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:56.225331 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
W20260928 08:59:56.225550 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:56.220616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36831 (local address 127.0.98.62:35473)
0928 08:59:56.220728 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:56.220730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:56.220816 (+ 86us) server_negotiation.cc:407] Connection header received
0928 08:59:56.220892 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:56.220893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:56.220907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:56.220959 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:56.221274 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.221772 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.222577 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.222787 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.222909 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:56.225466 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36831: BlockingRecv error: recv got EOF from 127.0.98.3:36831 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 08:59:56 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:56 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:56.960675 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:56.956034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:39125)
0928 08:59:56.956219 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:56.956230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:56.956292 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:56.956428 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:56.956428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:56.956595 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:56.956711 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:56.956713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.957357 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.957358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:56.957883 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:56.957884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.958372 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.958373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:56.958437 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:56.958870 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:56.958874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:56.960599 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
W20260928 08:59:56.960830 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:56.956116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39125 (local address 127.0.98.62:35473)
0928 08:59:56.956220 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:56.956222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:56.956299 (+ 77us) server_negotiation.cc:407] Connection header received
0928 08:59:56.956337 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:56.956337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:56.956347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:56.956404 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:56.956787 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.957285 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.957955 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:56.958211 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:56.958302 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:56.960737 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39125: BlockingRecv error: recv got EOF from 127.0.98.1:39125 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
Sep 28 08:59:57 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:57 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:57.085978 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:57.081311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:44793)
0928 08:59:57.081454 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:57.081470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:57.081529 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:57.081659 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:57.081660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:57.081818 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:57.081866 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:57.081868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.082504 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.082505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:57.083037 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:57.083039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.083351 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.083352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:57.083392 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:57.083966 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:57.083972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:57.085914 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260928 08:59:57.086105 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:57.081382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44793 (local address 127.0.98.62:35473)
0928 08:59:57.081455 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:57.081457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:57.081513 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:59:57.081572 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:57.081573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:57.081583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:57.081637 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:57.081929 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.082429 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.083112 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.083288 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.083337 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:57.086027 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44793: BlockingRecv error: recv got EOF from 127.0.98.2:44793 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 08:59:57 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:57 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:57.230827 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:57.226034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:51281)
0928 08:59:57.226152 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:57.226169 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:57.226262 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:57.226411 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:57.226412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:57.226658 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:57.226707 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:57.226709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.227383 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.227384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:57.227928 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:57.227930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.228349 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.228350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:57.228395 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:57.228949 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:57.228953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:57.230769 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260928 08:59:57.230964 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:57.226105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51281 (local address 127.0.98.62:35473)
0928 08:59:57.226153 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:57.226155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:57.226285 (+ 130us) server_negotiation.cc:407] Connection header received
0928 08:59:57.226312 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:57.226313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:57.226328 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:57.226387 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:57.226776 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.227293 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.228058 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.228285 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.228341 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:57.230880 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51281: BlockingRecv error: recv got EOF from 127.0.98.3:51281 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 08:59:57 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:57 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:57.972702 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:57.961172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:47081)
0928 08:59:57.961336 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:57.961348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:57.961416 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:57.961619 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:57.961620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:57.961786 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:57.961834 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:57.961836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.962503 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.962504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:57.963029 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:57.963030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.963361 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.963362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:57.963403 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:57.970527 (+ 7124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:57.970534 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:57.972579 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":55,"threads_started":1}
W20260928 08:59:57.972920 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:57.961377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47081 (local address 127.0.98.62:35473)
0928 08:59:57.961495 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:57.961497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:57.961505 (+ 8us) server_negotiation.cc:407] Connection header received
0928 08:59:57.961526 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:57.961527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:57.961536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:57.961593 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:57.961906 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.962413 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.963106 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:57.963293 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:57.963341 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:57.972844 (+ 9503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47081: BlockingRecv error: recv got EOF from 127.0.98.1:47081 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
Sep 28 08:59:58 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:58 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:58.091305 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:58.086514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:47163)
0928 08:59:58.086684 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:58.086696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:58.086764 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:58.086907 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:58.086907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:58.087088 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:58.087134 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:58.087136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.087832 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.087833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:58.088364 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:58.088366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.088808 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.088809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:58.088867 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:58.089320 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:58.089324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:58.091207 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
W20260928 08:59:58.091352 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:58.086577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47163 (local address 127.0.98.62:35473)
0928 08:59:58.086645 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:58.086647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:58.086745 (+ 98us) server_negotiation.cc:407] Connection header received
0928 08:59:58.086822 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:58.086823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:58.086835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:58.086886 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:58.087238 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.087738 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.088455 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.088715 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.088807 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:58.091284 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47163: BlockingRecv error: recv got EOF from 127.0.98.2:47163 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 08:59:58 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:58 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:58.236030 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:58.231258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:60189)
0928 08:59:58.231392 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:58.231409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:58.231493 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:58.231636 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:58.231637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:58.231811 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:58.231855 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:58.231857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.232542 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.232543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:58.233145 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:58.233147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.233534 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.233535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:58.233606 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:58.234055 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:58.234059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:58.235934 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
W20260928 08:59:58.236088 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:58.231416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60189 (local address 127.0.98.62:35473)
0928 08:59:58.231477 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:58.231479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:58.231489 (+ 10us) server_negotiation.cc:407] Connection header received
0928 08:59:58.231559 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:58.231560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:58.231575 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:58.231629 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:58.231938 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.232446 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.233246 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.233441 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.233533 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:58.236012 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60189: BlockingRecv error: recv got EOF from 127.0.98.3:60189 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 08:59:58 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:58 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:58.978741 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:58.973218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:60747)
0928 08:59:58.973427 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:58.973455 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:58.973532 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:58.973743 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:58.973745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:58.973960 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:58.974019 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:58.974021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.975022 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.975023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:58.975617 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:58.975619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.976110 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.976111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:58.976166 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:58.976607 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:58.976611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:58.978666 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
W20260928 08:59:58.978845 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:58.973332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60747 (local address 127.0.98.62:35473)
0928 08:59:58.973425 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:58.973427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:58.973576 (+ 149us) server_negotiation.cc:407] Connection header received
0928 08:59:58.973607 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:58.973607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:58.973620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:58.973715 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:58.974142 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.974925 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.975713 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:58.975980 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:58.976046 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:58.978769 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60747: BlockingRecv error: recv got EOF from 127.0.98.1:60747 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":43,"threads_started":1}
Sep 28 08:59:59 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:59 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:59.096936 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:59.091928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:36277)
0928 08:59:59.092121 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:59.092135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:59.092232 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:59.092366 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:59.092367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:59.092552 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:59.092598 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:59.092600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.093357 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.093358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:59.093899 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:59.093900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.094325 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.094326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:59.094384 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:59.094852 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:59.094856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:59.096865 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
W20260928 08:59:59.097054 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:59.092024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36277 (local address 127.0.98.62:35473)
0928 08:59:59.092118 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:59.092120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:59.092237 (+ 117us) server_negotiation.cc:407] Connection header received
0928 08:59:59.092291 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:59.092292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:59.092306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:59.092351 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:59.092750 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.093258 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.093999 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.094226 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.094323 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:59.096974 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36277: BlockingRecv error: recv got EOF from 127.0.98.2:36277 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 08:59:59 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:59 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:59:59.241281 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:59.236611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46699 (local address 127.0.98.62:35473)
0928 08:59:59.236721 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:59.236723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:59.236764 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:59:59.236840 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:59.236841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:59.236854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:59.236930 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:59.237293 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.237797 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.238518 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.238743 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.238826 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:59.241223 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46699: BlockingRecv error: recv got EOF from 127.0.98.3:46699 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:59:59.241281 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:59.236520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:46699)
0928 08:59:59.236694 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:59.236707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:59.236784 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:59.236940 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:59.236941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:59.237117 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:59.237181 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:59.237183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.237885 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.237887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:59.238424 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:59.238425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.238823 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.238824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:59.238895 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:59.239371 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:59.239375 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:59.241139 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
Sep 28 08:59:59 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:59:59 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:59:59.983754 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:59:59.979236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:54967)
0928 08:59:59.979364 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:59:59.979375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 08:59:59.979439 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:59:59.979621 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:59:59.979622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:59:59.979802 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:59:59.979851 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:59.979853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.980495 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.980496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:59.981127 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:59:59.981129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.981484 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.981485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:59:59.981525 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:59.981963 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:59:59.981967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:59:59.983695 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
W20260928 08:59:59.983842 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:59:59.979316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54967 (local address 127.0.98.62:35473)
0928 08:59:59.979457 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 08:59:59.979459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 08:59:59.979471 (+ 12us) server_negotiation.cc:407] Connection header received
0928 08:59:59.979508 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:59:59.979509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:59:59.979519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:59:59.979577 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:59:59.979930 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.980419 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.981204 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:59:59.981397 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:59:59.981479 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:59:59.983790 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54967: BlockingRecv error: recv got EOF from 127.0.98.1:54967 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
Sep 28 09:00:00 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:00 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:00.102654 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:00.097478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:58761)
0928 09:00:00.097654 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:00.097667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:00.097738 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:00.097885 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:00.097886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:00.098078 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:00.098124 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:00.098125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.098804 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.098805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:00.099419 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:00.099421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.099901 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.099902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:00.099961 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:00.100448 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:00.100453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:00.102544 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
W20260928 09:00:00.102705 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:00.097599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58761 (local address 127.0.98.62:35473)
0928 09:00:00.097652 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:00.097654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:00.097719 (+ 65us) server_negotiation.cc:407] Connection header received
0928 09:00:00.097793 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:00.097794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:00.097807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:00.097880 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:00.098212 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.098710 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.099513 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.099798 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.099907 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:00.102634 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58761: BlockingRecv error: recv got EOF from 127.0.98.2:58761 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 09:00:00 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:00 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:00.247434 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:00.241825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:50207)
0928 09:00:00.241986 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:00.242005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:00.242055 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:00.242333 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:00.242334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:00.242531 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:00.242577 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:00.242579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.243280 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.243281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:00.243822 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:00.243824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.244408 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.244409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:00.244483 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:00.245130 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:00.245136 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:00.247331 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
W20260928 09:00:00.247530 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:00.241895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50207 (local address 127.0.98.62:35473)
0928 09:00:00.242200 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:00.242202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:00.242210 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:00:00.242239 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:00.242239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:00.242252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:00.242323 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:00.242663 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.243179 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.243961 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.244279 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.244355 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:00.247448 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50207: BlockingRecv error: recv got EOF from 127.0.98.3:50207 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 09:00:00 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:00 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:00.989156 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:00.984231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:54591)
0928 09:00:00.984422 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:00.984432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:00.984499 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:00.984615 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:00.984616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:00.984820 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:00.984868 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:00.984871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.985566 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.985567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:00.986094 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:00.986096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.986591 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.986592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:00.986656 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:00.987197 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:00.987202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:00.989051 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":53,"threads_started":1}
W20260928 09:00:00.989226 9626 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:00.984310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54591 (local address 127.0.98.62:35473)
0928 09:00:00.984420 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:00.984423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:00.984477 (+ 54us) server_negotiation.cc:407] Connection header received
0928 09:00:00.984552 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:00.984553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:00.984563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:00.984610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:00.984965 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.985476 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.986177 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:00.986391 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:00.986587 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:00.989148 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54591: BlockingRecv error: recv got EOF from 127.0.98.1:54591 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
Sep 28 09:00:01 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:01 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:01.108079 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:01.103222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:50221)
0928 09:00:01.103396 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:01.103430 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:01.103504 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:01.103639 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:01.103639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:01.103798 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:01.103848 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:01.103849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.104467 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.104468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:01.105100 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:01.105103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.105573 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.105574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:01.105617 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:01.106165 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:01.106171 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:01.107977 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
W20260928 09:00:01.108109 9626 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:01.103387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50221 (local address 127.0.98.62:35473)
0928 09:00:01.103458 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:01.103461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:01.103490 (+ 29us) server_negotiation.cc:407] Connection header received
0928 09:00:01.103553 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:01.103554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:01.103564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:01.103615 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:01.103916 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.104397 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.105185 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.105484 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.105554 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:01.108051 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50221: BlockingRecv error: recv got EOF from 127.0.98.2:50221 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 09:00:01 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:01 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:01.253026 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:01.248192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:55827)
0928 09:00:01.248340 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:01.248358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:01.248413 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:01.248535 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:01.248536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:01.248765 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:01.248815 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:01.248818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.249465 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.249466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:01.250004 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:01.250006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.250357 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.250358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:01.250397 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:01.250889 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:01.250894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:01.252964 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
W20260928 09:00:01.253057 9626 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:01.248257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55827 (local address 127.0.98.62:35473)
0928 09:00:01.248347 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:01.248350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:01.248420 (+ 70us) server_negotiation.cc:407] Connection header received
0928 09:00:01.248452 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:01.248453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:01.248466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:01.248525 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:01.248883 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.249388 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.250098 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.250283 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.250341 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:01.253000 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55827: BlockingRecv error: recv got EOF from 127.0.98.3:55827 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 09:00:01 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:01 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:01.994855 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:01.989732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:38953)
0928 09:00:01.989859 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:01.989883 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:01.989952 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:01.990255 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:01.990256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:01.990452 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:01.990506 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:01.990508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.991266 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.991266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:01.991800 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:01.991802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.992250 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.992251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:01.992292 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:01.992860 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:01.992864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:01.994797 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
W20260928 09:00:01.995007 9630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:01.989807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38953 (local address 127.0.98.62:35473)
0928 09:00:01.989905 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:01.989907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:01.989933 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:00:01.990014 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:01.990014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:01.990026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:01.990123 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:01.990590 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.991187 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.991900 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:01.992169 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:01.992238 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:01.994916 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38953: BlockingRecv error: recv got EOF from 127.0.98.1:38953 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
Sep 28 09:00:02 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:02 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:02.113426 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:02.108682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:37811)
0928 09:00:02.108816 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:02.108833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:02.108920 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:02.109088 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:02.109089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:02.109266 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:02.109312 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:02.109314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.110011 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.110012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:02.110539 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:02.110541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.110897 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.110898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:02.110940 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:02.111511 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:02.111516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:02.113364 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260928 09:00:02.113536 9630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:02.108748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37811 (local address 127.0.98.62:35473)
0928 09:00:02.108812 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:02.108814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:02.108906 (+ 92us) server_negotiation.cc:407] Connection header received
0928 09:00:02.108994 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:02.108995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:02.109007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:02.109084 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:02.109389 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.109888 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.110639 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.110816 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.110870 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:02.113460 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37811: BlockingRecv error: recv got EOF from 127.0.98.2:37811 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 09:00:02 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:02 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:02.258422 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:02.253549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.3:48701)
0928 09:00:02.253723 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:02.253735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:02.253829 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:02.253994 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:02.253995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:02.254185 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:02.254233 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:02.254235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.254911 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.254912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:02.255445 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:02.255446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.255847 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.255848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:02.255904 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:02.256431 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:02.256436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:02.258322 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
W20260928 09:00:02.258473 9630 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:02.253635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48701 (local address 127.0.98.62:35473)
0928 09:00:02.253688 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:02.253690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:02.253807 (+ 117us) server_negotiation.cc:407] Connection header received
0928 09:00:02.253883 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:02.253884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:02.253899 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:02.253982 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:02.254325 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.254819 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.255575 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.255768 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.255850 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:02.258408 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48701: BlockingRecv error: recv got EOF from 127.0.98.3:48701 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 09:00:03 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:03 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:03.008841 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:02.995416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.1:34045)
0928 09:00:02.995525 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:02.995536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:02.995596 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:02.995803 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:02.995804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:02.995958 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:02.996005 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:02.996007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.997022 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.997023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:02.997620 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:02.997622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.998109 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.998110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:02.998155 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.006801 (+ 8646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:03.006808 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:03.008764 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
W20260928 09:00:03.008980 9634 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:02.995494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34045 (local address 127.0.98.62:35473)
0928 09:00:02.995664 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:02.995666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:02.995681 (+ 15us) server_negotiation.cc:407] Connection header received
0928 09:00:02.995705 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:02.995706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:02.995716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:02.995785 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:02.996130 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.996929 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.997706 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:02.998033 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:02.998106 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.008887 (+ 10781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34045: BlockingRecv error: recv got EOF from 127.0.98.1:34045 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
Sep 28 09:00:03 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 09:00:03 dist-test-slave-286n krb5kdc[8574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 09:00:03.119065 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.114026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35473 (local address 127.0.98.2:49269)
0928 09:00:03.114341 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:03.114360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:03.114454 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:03.114626 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:03.114627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:03.114803 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:03.114851 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.114853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.115545 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.115547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.116101 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.116103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.116530 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.116531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.116595 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.117090 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:03.117095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:03.118992 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35473: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":162,"threads_started":1}
W20260928 09:00:03.119136 9634 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:03.114136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49269 (local address 127.0.98.62:35473)
0928 09:00:03.114321 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.114323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.114433 (+ 110us) server_negotiation.cc:407] Connection header received
0928 09:00:03.114550 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.114551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.114563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.114611 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.114946 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.115456 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.116209 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.116428 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.116520 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.119090 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49269: BlockingRecv error: recv got EOF from 127.0.98.2:49269 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 09:00:03.174041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9157
I20260928 09:00:03.178694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9309
I20260928 09:00:03.183008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9442
I20260928 09:00:03.188042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9087
2026-09-28T09:00:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17555 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T09:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:00:03Z Disabled control of system clock
I20260928 09:00:03.216089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41115
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 09:00:03.296216 9639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:03.296437 9639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:03.296463 9639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:03.297658 9639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:00:03.297709 9639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:03.297722 9639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:03.297739 9639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:00:03.297758 9639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:00:03.299080 9639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:03.299289 9639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:03.299533 9639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:03.305294 9647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.305302 9645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:03.305358 9639 server_base.cc:1034] running on GCE node
W20260928 09:00:03.305294 9644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:03.305886 9639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:03.306200 9639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:03.307344 9639 hybrid_clock.cc:648] HybridClock initialized: now 1790586003307324 us; error 38 us; skew 500 ppm
I20260928 09:00:03.308753 9639 webserver.cc:460] Webserver started at http://127.0.98.62:39187/ using document root <none> and password file <none>
I20260928 09:00:03.309005 9639 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:03.309077 9639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:03.309243 9639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:03.310372 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "975f3e79be6f4f258aa6589c4c802dfb"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "5955b31dce983cffe63659eca5197128"
server_key_iv: "04b5acf221ad054b47129d57d631d292"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.310678 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "975f3e79be6f4f258aa6589c4c802dfb"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "5955b31dce983cffe63659eca5197128"
server_key_iv: "04b5acf221ad054b47129d57d631d292"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.311930 9639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:00:03.312759 9653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.312981 9639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:03.313073 9639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "975f3e79be6f4f258aa6589c4c802dfb"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "5955b31dce983cffe63659eca5197128"
server_key_iv: "04b5acf221ad054b47129d57d631d292"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.313139 9639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:03.323858 9639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:03.324137 9639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:03.324246 9639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:03.328097 9639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41115
I20260928 09:00:03.328166 9705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41115 every 8 connection(s)
I20260928 09:00:03.328500 9639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:00:03.329327 9706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:03.330492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9639
I20260928 09:00:03.330569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:00:03.330799 392 external_mini_cluster.cc:1428] Setting key 737f9937e4b216d5cc1c73c68f335b02
I20260928 09:00:03.333127 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb: Bootstrap starting.
I20260928 09:00:03.333845 9706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:03.334134 9706 log.cc:824] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:03.334851 9706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb: No bootstrap required, opened a new log
I20260928 09:00:03.337054 9706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975f3e79be6f4f258aa6589c4c802dfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } }
I20260928 09:00:03.337296 9706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:03.337363 9706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975f3e79be6f4f258aa6589c4c802dfb, State: Initialized, Role: FOLLOWER
I20260928 09:00:03.337519 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.331354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.0.98.62:41115)
0928 09:00:03.331621 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.331631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.331672 (+ 41us) server_negotiation.cc:407] Connection header received
0928 09:00:03.332232 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.332242 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.332325 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.332476 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.332747 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.334079 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.334986 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.335589 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.335732 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.336691 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:03.336696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:03.336697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:03.336926 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:03.336968 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:03.336982 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:03.337055 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:03.337210 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 09:00:03.337292 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260928 09:00:03.337486 9706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975f3e79be6f4f258aa6589c4c802dfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } }
I20260928 09:00:03.337572 9706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:00:03.337605 9706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:00:03.337651 9706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:03.338248 9706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975f3e79be6f4f258aa6589c4c802dfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } }
I20260928 09:00:03.338378 9706 leader_election.cc:302] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 975f3e79be6f4f258aa6589c4c802dfb; no voters:
I20260928 09:00:03.338564 9706 leader_election.cc:288] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:00:03.338598 9711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:00:03.338766 9711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [term 1 LEADER]: Becoming Leader. State: Replica: 975f3e79be6f4f258aa6589c4c802dfb, State: Running, Role: LEADER
I20260928 09:00:03.338825 9706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:00:03.338902 9711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975f3e79be6f4f258aa6589c4c802dfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } }
I20260928 09:00:03.339399 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "975f3e79be6f4f258aa6589c4c802dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975f3e79be6f4f258aa6589c4c802dfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } }
I20260928 09:00:03.339414 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 975f3e79be6f4f258aa6589c4c802dfb. Latest consensus state: current_term: 1 leader_uuid: "975f3e79be6f4f258aa6589c4c802dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "975f3e79be6f4f258aa6589c4c802dfb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41115 } } }
I20260928 09:00:03.339504 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:03.339511 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:03.339710 9720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:00:03.340209 9720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:00:03.341741 9720 catalog_manager.cc:1347] Generated new cluster ID: b53d9b61d62d443b8533203613485fa0
I20260928 09:00:03.341815 9720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:00:03.351008 9720 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:00:03.351740 9720 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:00:03.361207 9720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 975f3e79be6f4f258aa6589c4c802dfb: Generated new TSK 0
I20260928 09:00:03.361421 9720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:00:03.368580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41115
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 09:00:03.469986 9730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:03.470228 9730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:03.470268 9730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:03.471448 9730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:03.471519 9730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:03.471563 9730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:00:03.473063 9730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:03.473359 9730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:03.473722 9730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:03.478936 9736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.478984 9735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.478981 9738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:03.479455 9730 server_base.cc:1034] running on GCE node
I20260928 09:00:03.479694 9730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:03.479946 9730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:03.481110 9730 hybrid_clock.cc:648] HybridClock initialized: now 1790586003481092 us; error 46 us; skew 500 ppm
I20260928 09:00:03.482402 9730 webserver.cc:460] Webserver started at http://127.0.98.1:37989/ using document root <none> and password file <none>
I20260928 09:00:03.482642 9730 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:03.482710 9730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:03.482878 9730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:03.483949 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "8680808be63a478c996578fb1d1e5de9"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "44def0d815ed5136e766eb0255353a8f"
server_key_iv: "171fd7bf1f644eb1c135a18c1d9586cb"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.484323 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8680808be63a478c996578fb1d1e5de9"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "44def0d815ed5136e766eb0255353a8f"
server_key_iv: "171fd7bf1f644eb1c135a18c1d9586cb"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.485836 9730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:00:03.486652 9744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.486827 9730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:03.486909 9730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "8680808be63a478c996578fb1d1e5de9"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "44def0d815ed5136e766eb0255353a8f"
server_key_iv: "171fd7bf1f644eb1c135a18c1d9586cb"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.486984 9730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:03.494880 9730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:03.495215 9730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:03.495384 9730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:03.495613 9730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:03.495956 9730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:03.496016 9730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.496060 9730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:03.496086 9730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.501803 9730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33917
I20260928 09:00:03.501895 9857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33917 every 8 connection(s)
I20260928 09:00:03.502228 9730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:00:03.504052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9730
I20260928 09:00:03.504122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:00:03.504341 392 external_mini_cluster.cc:1428] Setting key 6ef4daf23fc77b1ccd4cc1287f1f10a5
I20260928 09:00:03.505412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41115
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 09:00:03.508876 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.503070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38667 (local address 127.0.98.62:41115)
0928 09:00:03.503161 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.503164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.503408 (+ 244us) server_negotiation.cc:407] Connection header received
0928 09:00:03.504008 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.504009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.504023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.504100 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.505172 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.505943 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.507186 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.507384 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.507477 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.508439 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:03.508441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:03.508441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:03.508447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:03.508465 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:03.508470 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:03.508534 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:03.508731 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 09:00:03.508768 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:00:03.509121 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.502991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41115 (local address 127.0.98.1:38667)
0928 09:00:03.503267 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:03.503302 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:03.503880 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:03.504226 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:03.504230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:03.504241 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:03.504959 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.504966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.506070 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.506136 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.506978 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.506981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.507484 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.507485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.507640 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.508279 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:00:03.508280 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:03.508309 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:00:03.508333 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:03.508579 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:03.508587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:03.508590 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:00:03.508708 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 09:00:03.508835 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260928 09:00:03.509696 9858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41115
I20260928 09:00:03.509831 9858 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:03.510031 9858 heartbeater.cc:511] Master 127.0.98.62:41115 requested a full tablet report, sending...
I20260928 09:00:03.510634 9670 ts_manager.cc:194] Registered new tserver with Master: 8680808be63a478c996578fb1d1e5de9 (127.0.98.1:33917)
I20260928 09:00:03.511313 9670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38667
W20260928 09:00:03.587339 9861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:03.587597 9861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:03.587635 9861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:03.588819 9861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:03.588896 9861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:03.588938 9861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:00:03.590361 9861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:03.590662 9861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:03.590976 9861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:03.596310 9867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.596310 9866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.596239 9869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:03.596541 9861 server_base.cc:1034] running on GCE node
I20260928 09:00:03.596835 9861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:03.597144 9861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:03.598361 9861 hybrid_clock.cc:648] HybridClock initialized: now 1790586003598330 us; error 38 us; skew 500 ppm
I20260928 09:00:03.599854 9861 webserver.cc:460] Webserver started at http://127.0.98.2:44275/ using document root <none> and password file <none>
I20260928 09:00:03.600109 9861 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:03.600188 9861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:03.600347 9861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:03.601521 9861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbe51902baff4279b724a9fc83d0f269"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "1eeb09759d2d5493d51779f5cfbdaf1d"
server_key_iv: "48c9f77b53f75cf6aebe63002c6d7230"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.601897 9861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbe51902baff4279b724a9fc83d0f269"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "1eeb09759d2d5493d51779f5cfbdaf1d"
server_key_iv: "48c9f77b53f75cf6aebe63002c6d7230"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.603256 9861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:00:03.604053 9875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.604252 9861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:03.604369 9861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "bbe51902baff4279b724a9fc83d0f269"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "1eeb09759d2d5493d51779f5cfbdaf1d"
server_key_iv: "48c9f77b53f75cf6aebe63002c6d7230"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.604445 9861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:03.613484 9861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:03.613818 9861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:03.613981 9861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:03.614205 9861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:03.614552 9861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:03.614643 9861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.614712 9861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:03.614744 9861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.620903 9861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38121
I20260928 09:00:03.620975 9988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38121 every 8 connection(s)
I20260928 09:00:03.621313 9861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:00:03.626819 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.622100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38069 (local address 127.0.98.62:41115)
0928 09:00:03.622169 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.622171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.622477 (+ 306us) server_negotiation.cc:407] Connection header received
0928 09:00:03.622953 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.622954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.622964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.623015 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.623726 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.624302 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.625383 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.625574 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.625645 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.626488 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:03.626490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:03.626490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:03.626494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:03.626506 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:03.626510 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:03.626568 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:03.626695 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 09:00:03.626728 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:00:03.626991 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.622085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41115 (local address 127.0.98.2:38069)
0928 09:00:03.622354 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:03.622387 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:03.622869 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:03.623069 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:03.623072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:03.623080 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:03.623599 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.623602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.624393 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.624393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.625267 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.625269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.625660 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.625660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.625766 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.626348 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:00:03.626348 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:03.626377 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:00:03.626398 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:03.626577 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:03.626581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:03.626584 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:00:03.626644 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 09:00:03.626724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":158,"threads_started":1}
I20260928 09:00:03.627475 9989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41115
I20260928 09:00:03.627563 9989 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:03.627701 9989 heartbeater.cc:511] Master 127.0.98.62:41115 requested a full tablet report, sending...
I20260928 09:00:03.628106 9670 ts_manager.cc:194] Registered new tserver with Master: bbe51902baff4279b724a9fc83d0f269 (127.0.98.2:38121)
I20260928 09:00:03.628513 9670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38069
I20260928 09:00:03.630360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9861
I20260928 09:00:03.630436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:00:03.630641 392 external_mini_cluster.cc:1428] Setting key 34c1235fb7077eb9ff3d53dfe5978537
I20260928 09:00:03.631556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41115
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 09:00:03.713603 9992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:03.713869 9992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:03.713907 9992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:03.715123 9992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:03.715204 9992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:03.715257 9992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:00:03.716925 9992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:03.717206 9992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:03.717545 9992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:03.723017 10000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.723028 9997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.723017 9998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:03.723263 9992 server_base.cc:1034] running on GCE node
I20260928 09:00:03.723531 9992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:03.723855 9992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:03.725035 9992 hybrid_clock.cc:648] HybridClock initialized: now 1790586003725015 us; error 42 us; skew 500 ppm
I20260928 09:00:03.726528 9992 webserver.cc:460] Webserver started at http://127.0.98.3:43839/ using document root <none> and password file <none>
I20260928 09:00:03.726809 9992 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:03.726886 9992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:03.727041 9992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:03.728148 9992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "bbba38b2ae5b4f55bd5b25118eacd155"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "d7079bc03b65964499b985e8971c0d9e"
server_key_iv: "dcad281be2d051622ca778f8e56c409f"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.728534 9992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bbba38b2ae5b4f55bd5b25118eacd155"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "d7079bc03b65964499b985e8971c0d9e"
server_key_iv: "dcad281be2d051622ca778f8e56c409f"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.729902 9992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:03.730820 10006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.731005 9992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:03.731102 9992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "bbba38b2ae5b4f55bd5b25118eacd155"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "d7079bc03b65964499b985e8971c0d9e"
server_key_iv: "dcad281be2d051622ca778f8e56c409f"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.731191 9992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:03.748528 9992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:03.748919 9992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:03.749092 9992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:03.749360 9992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:03.749725 9992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:03.749785 9992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.749827 9992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:03.749854 9992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.755635 9992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32813
I20260928 09:00:03.755679 10119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32813 every 8 connection(s)
I20260928 09:00:03.756009 9992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:00:03.756598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 9992
I20260928 09:00:03.756742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:00:03.756981 392 external_mini_cluster.cc:1428] Setting key fd2db1ea114fbc6eb393afc2bd3627b4
I20260928 09:00:03.761754 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.756797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43987 (local address 127.0.98.62:41115)
0928 09:00:03.756850 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.756853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.757145 (+ 292us) server_negotiation.cc:407] Connection header received
0928 09:00:03.757949 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.757950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.757963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.758024 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.758712 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.759209 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.760169 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.760359 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.760416 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.761525 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:03.761526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:03.761526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:03.761530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:03.761544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:03.761548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:03.761591 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:03.761672 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 09:00:03.761692 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:00:03.762013 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.756779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41115 (local address 127.0.98.3:43987)
0928 09:00:03.756993 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:03.757045 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:03.757793 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:03.758096 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:03.758100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:03.758111 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:03.758631 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.758636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.759294 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.759295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.760089 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:03.760090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:03.760485 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:03.760487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:03.760615 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:03.761390 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:00:03.761391 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:03.761421 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:00:03.761446 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:03.761602 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:03.761605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:03.761607 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:00:03.761660 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 09:00:03.761742 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260928 09:00:03.762588 10120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41115
I20260928 09:00:03.762686 10120 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:03.762904 10120 heartbeater.cc:511] Master 127.0.98.62:41115 requested a full tablet report, sending...
I20260928 09:00:03.763376 9670 ts_manager.cc:194] Registered new tserver with Master: bbba38b2ae5b4f55bd5b25118eacd155 (127.0.98.3:32813)
I20260928 09:00:03.763715 9670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43987
I20260928 09:00:03.769024 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 09:00:03.771214 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:03.770496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.98.62:41115)
0928 09:00:03.770614 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.770617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.770772 (+ 155us) server_negotiation.cc:407] Connection header received
0928 09:00:03.770849 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.770850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.770861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.770946 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.771107 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35820: BlockingRecv error: recv got EOF from 127.0.0.1:35820 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 09:00:03.771373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9730
I20260928 09:00:03.776319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9861
I20260928 09:00:03.781014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9992
I20260928 09:00:03.785506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 9639
2026-09-28T09:00:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (596 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T09:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:00:03Z Disabled control of system clock
I20260928 09:00:03.812696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41349
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:00:03.894460 10131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:03.894677 10131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:03.894699 10131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:03.895823 10131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:00:03.895859 10131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:03.895872 10131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:03.895884 10131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:00:03.895896 10131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:00:03.897317 10131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:03.897524 10131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:03.897789 10131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:00:03.902920 10131 server_base.cc:1034] running on GCE node
W20260928 09:00:03.902860 10139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.902895 10136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:03.902864 10137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:03.903463 10131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:03.903774 10131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:03.904983 10131 hybrid_clock.cc:648] HybridClock initialized: now 1790586003904968 us; error 57 us; skew 500 ppm
I20260928 09:00:03.906410 10131 webserver.cc:460] Webserver started at http://127.0.98.62:39611/ using document root <none> and password file <none>
I20260928 09:00:03.906723 10131 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:03.906800 10131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:03.906944 10131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:03.908047 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "4591664ea7704a9b90b831754a3287c9"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "ad31d121bb1952cd6c6b9f0a305c97d0"
server_key_iv: "e2a36efae3acce6e104755b143f2754e"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.908437 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "4591664ea7704a9b90b831754a3287c9"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "ad31d121bb1952cd6c6b9f0a305c97d0"
server_key_iv: "e2a36efae3acce6e104755b143f2754e"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.909869 10131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:03.910665 10145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:03.910866 10131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:00:03.910970 10131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "4591664ea7704a9b90b831754a3287c9"
format_stamp: "Formatted at 2026-09-28 09:00:03 on dist-test-slave-286n"
server_key: "ad31d121bb1952cd6c6b9f0a305c97d0"
server_key_iv: "e2a36efae3acce6e104755b143f2754e"
server_key_version: "encryptionkey@0"
I20260928 09:00:03.911068 10131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:03.914659 10131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:03.914966 10131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:03.915120 10131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:03.919054 10131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41349
I20260928 09:00:03.919085 10197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41349 every 8 connection(s)
I20260928 09:00:03.919446 10131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:00:03.920331 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:03.923250 10198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9: Bootstrap starting.
I20260928 09:00:03.923897 10198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:03.924216 10198 log.cc:824] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:03.925043 10198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9: No bootstrap required, opened a new log
I20260928 09:00:03.926668 10198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4591664ea7704a9b90b831754a3287c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260928 09:00:03.926813 10198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:03.926837 10198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4591664ea7704a9b90b831754a3287c9, State: Initialized, Role: FOLLOWER
I20260928 09:00:03.927008 10198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4591664ea7704a9b90b831754a3287c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260928 09:00:03.927098 10198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:00:03.927080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10131
I20260928 09:00:03.927131 10198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:00:03.927172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:00:03.927191 10198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:03.927376 392 external_mini_cluster.cc:1428] Setting key 871bfb0b913378e74641b5201a76bdfa
I20260928 09:00:03.927951 10198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4591664ea7704a9b90b831754a3287c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260928 09:00:03.928068 10198 leader_election.cc:302] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4591664ea7704a9b90b831754a3287c9; no voters:
I20260928 09:00:03.928259 10198 leader_election.cc:288] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:00:03.928333 10203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:00:03.928478 10203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4591664ea7704a9b90b831754a3287c9, State: Running, Role: LEADER
I20260928 09:00:03.928562 10198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:00:03.928571 10203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4591664ea7704a9b90b831754a3287c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } }
I20260928 09:00:03.929209 10204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4591664ea7704a9b90b831754a3287c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4591664ea7704a9b90b831754a3287c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } } }
I20260928 09:00:03.929306 10204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:03.929524 10212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:00:03.929533 10205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4591664ea7704a9b90b831754a3287c9. Latest consensus state: current_term: 1 leader_uuid: "4591664ea7704a9b90b831754a3287c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4591664ea7704a9b90b831754a3287c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41349 } } }
I20260928 09:00:03.930033 10205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9 [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:03.930084 10212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:00:03.930557 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:03.927910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.0.98.62:41349)
0928 09:00:03.928209 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:03.928219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:03.928254 (+ 35us) server_negotiation.cc:407] Connection header received
0928 09:00:03.928382 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:03.928390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:03.928447 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:03.929216 (+ 769us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:03.929842 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:03.929846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:03.929846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:03.929851 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:03.929900 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:03.929914 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:03.930207 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:03.930241 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 09:00:03.930301 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":134,"threads_started":1}
I20260928 09:00:03.931929 10212 catalog_manager.cc:1347] Generated new cluster ID: 79180947dffd4ee38bd4bdf441659d6d
I20260928 09:00:03.931980 10212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:00:03.951887 10212 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:00:03.952133 10212 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:00:03.958801 10212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4591664ea7704a9b90b831754a3287c9: Generated new TSK 0
I20260928 09:00:03.959025 10212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:00:03.966964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:00:04.050029 10222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:04.050289 10222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:04.050329 10222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:04.051476 10222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:04.051551 10222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:04.051596 10222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:00:04.052954 10222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:04.053226 10222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:04.053532 10222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:04.059275 10227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:04.059353 10228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:04.059352 10230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:04.059748 10222 server_base.cc:1034] running on GCE node
I20260928 09:00:04.059984 10222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:04.060231 10222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:04.061390 10222 hybrid_clock.cc:648] HybridClock initialized: now 1790586004061376 us; error 29 us; skew 500 ppm
I20260928 09:00:04.062681 10222 webserver.cc:460] Webserver started at http://127.0.98.1:43457/ using document root <none> and password file <none>
I20260928 09:00:04.062963 10222 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:04.063014 10222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:04.063189 10222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:04.064237 10222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "eb0ba5d6b6c14a1a9b2c4a42317cb295"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "76cc8b02f4e69a5bfb9e7af3895eb0eb"
server_key_iv: "662a9800856da59320bd1264f2fba919"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.064606 10222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0ba5d6b6c14a1a9b2c4a42317cb295"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "76cc8b02f4e69a5bfb9e7af3895eb0eb"
server_key_iv: "662a9800856da59320bd1264f2fba919"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.065914 10222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:04.066617 10236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.066799 10222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:00:04.066871 10222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "eb0ba5d6b6c14a1a9b2c4a42317cb295"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "76cc8b02f4e69a5bfb9e7af3895eb0eb"
server_key_iv: "662a9800856da59320bd1264f2fba919"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.066965 10222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:04.070367 10222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:04.070624 10222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:04.070765 10222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:04.070966 10222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:04.071270 10222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:04.071326 10222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.071367 10222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:04.071393 10222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.077239 10222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44577
I20260928 09:00:04.077289 10349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44577 every 8 connection(s)
I20260928 09:00:04.077589 10222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:00:04.080397 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:04.078264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57359 (local address 127.0.98.62:41349)
0928 09:00:04.078330 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:04.078332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:04.078560 (+ 228us) server_negotiation.cc:407] Connection header received
0928 09:00:04.079003 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:04.079004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:04.079011 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:04.079060 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:04.080230 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:04.080231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:04.080231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:04.080233 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:04.080244 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:04.080248 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:04.080290 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:04.080336 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 09:00:04.080349 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:00:04.080600 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:04.078248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41349 (local address 127.0.98.1:57359)
0928 09:00:04.078431 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:04.078465 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:04.078921 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:04.079117 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:04.079121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:04.079128 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:04.080104 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:00:04.080104 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:04.080130 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:00:04.080157 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:04.080290 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:04.080292 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:04.080294 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:00:04.080344 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 09:00:04.080390 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260928 09:00:04.081230 10350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41349
I20260928 09:00:04.081362 10350 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:04.081475 10350 heartbeater.cc:511] Master 127.0.98.62:41349 requested a full tablet report, sending...
I20260928 09:00:04.081909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10222
I20260928 09:00:04.081965 10162 ts_manager.cc:194] Registered new tserver with Master: eb0ba5d6b6c14a1a9b2c4a42317cb295 (127.0.98.1:44577)
I20260928 09:00:04.082013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:00:04.082207 392 external_mini_cluster.cc:1428] Setting key 5ce6a128deccb071d1b450d9a3749ac1
I20260928 09:00:04.083122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:00:04.165683 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:04.165932 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:04.165969 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:04.167120 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:04.167200 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:04.167246 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:00:04.168619 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:04.168953 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:04.169250 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:04.174506 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:04.174512 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:04.174573 10353 server_base.cc:1034] running on GCE node
W20260928 09:00:04.174526 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:04.174957 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:04.175233 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:04.176405 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790586004176404 us; error 40 us; skew 500 ppm
I20260928 09:00:04.177695 10353 webserver.cc:460] Webserver started at http://127.0.98.2:37059/ using document root <none> and password file <none>
I20260928 09:00:04.177960 10353 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:04.178033 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:04.178169 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:04.179284 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "145355a5178f48adb5d26fc7ebd2ceb0"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "98fb80e659d66690eaeb4f6d98e68d66"
server_key_iv: "b575d7b5d251c0bd5ade38c800714e2b"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.179661 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "145355a5178f48adb5d26fc7ebd2ceb0"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "98fb80e659d66690eaeb4f6d98e68d66"
server_key_iv: "b575d7b5d251c0bd5ade38c800714e2b"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.181020 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:00:04.181792 10367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.181962 10353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:00:04.182065 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "145355a5178f48adb5d26fc7ebd2ceb0"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "98fb80e659d66690eaeb4f6d98e68d66"
server_key_iv: "b575d7b5d251c0bd5ade38c800714e2b"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.182147 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:04.185541 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:04.185822 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:04.185968 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:04.186167 10353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:04.186457 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:04.186512 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.186550 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:04.186578 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.192658 10353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41043
I20260928 09:00:04.192761 10480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41043 every 8 connection(s)
I20260928 09:00:04.193075 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:00:04.196882 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:04.193827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43539 (local address 127.0.98.62:41349)
0928 09:00:04.193873 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:04.193875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:04.194139 (+ 264us) server_negotiation.cc:407] Connection header received
0928 09:00:04.194829 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:04.194830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:04.194836 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:04.194934 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:04.196559 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:04.196560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:04.196560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:04.196565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:04.196583 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:04.196587 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:04.196676 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:04.196768 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:00:04.196782 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:00:04.197103 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:04.193818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41349 (local address 127.0.98.2:43539)
0928 09:00:04.194019 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:04.194052 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:04.194631 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:04.195032 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:04.195036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:04.195047 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:04.196346 (+ 1299us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:00:04.196347 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:04.196388 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:00:04.196427 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:04.196682 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:04.196687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:04.196691 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:00:04.196779 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:00:04.196847 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260928 09:00:04.197687 10481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41349
I20260928 09:00:04.197796 10481 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:04.197979 10481 heartbeater.cc:511] Master 127.0.98.62:41349 requested a full tablet report, sending...
I20260928 09:00:04.198390 10162 ts_manager.cc:194] Registered new tserver with Master: 145355a5178f48adb5d26fc7ebd2ceb0 (127.0.98.2:41043)
I20260928 09:00:04.199039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10353
I20260928 09:00:04.199132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:00:04.199389 392 external_mini_cluster.cc:1428] Setting key b2d1aacc73fc4cbac0c16547b2cca74c
I20260928 09:00:04.200335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 09:00:04.283546 10484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:04.283811 10484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:04.283854 10484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:04.285154 10484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:04.285236 10484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:04.285282 10484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:00:04.286696 10484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:04.286989 10484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:04.287294 10484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:04.292313 10492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:04.292443 10489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:04.292347 10490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:04.292577 10484 server_base.cc:1034] running on GCE node
I20260928 09:00:04.292812 10484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:04.293072 10484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:04.294234 10484 hybrid_clock.cc:648] HybridClock initialized: now 1790586004294261 us; error 68 us; skew 500 ppm
I20260928 09:00:04.295506 10484 webserver.cc:460] Webserver started at http://127.0.98.3:34447/ using document root <none> and password file <none>
I20260928 09:00:04.295761 10484 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:04.295832 10484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:04.296005 10484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:04.297149 10484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f6573319c5b4eae85457dcb1b70bf90"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "5b27636ed1403c7b2c07b3a90ed2d8c3"
server_key_iv: "0b0cb893758523d8862681c5eb1629ae"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.297552 10484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f6573319c5b4eae85457dcb1b70bf90"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "5b27636ed1403c7b2c07b3a90ed2d8c3"
server_key_iv: "0b0cb893758523d8862681c5eb1629ae"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.298974 10484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:00:04.299754 10498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.299976 10484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:04.300086 10484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "5f6573319c5b4eae85457dcb1b70bf90"
format_stamp: "Formatted at 2026-09-28 09:00:04 on dist-test-slave-286n"
server_key: "5b27636ed1403c7b2c07b3a90ed2d8c3"
server_key_iv: "0b0cb893758523d8862681c5eb1629ae"
server_key_version: "encryptionkey@0"
I20260928 09:00:04.300164 10484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:04.303583 10484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:04.303843 10484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:04.303983 10484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:04.304188 10484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:04.304538 10484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:04.304590 10484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.304628 10484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:04.304687 10484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:04.310524 10484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43221
I20260928 09:00:04.310570 10611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43221 every 8 connection(s)
I20260928 09:00:04.310911 10484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:00:04.314181 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:04.311669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44073 (local address 127.0.98.62:41349)
0928 09:00:04.311733 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:04.311735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:04.311995 (+ 260us) server_negotiation.cc:407] Connection header received
0928 09:00:04.312501 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:04.312502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:04.312509 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:04.312566 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:04.313886 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:04.313888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:04.313888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:00:04.313893 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:04.313908 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:00:04.313913 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:04.313988 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:04.314091 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 09:00:04.314109 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:00:04.314402 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:04.311652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41349 (local address 127.0.98.3:44073)
0928 09:00:04.311844 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:04.311893 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:04.312386 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:04.312670 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:04.312694 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:04.312705 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:04.313711 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:00:04.313712 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:04.313740 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:00:04.313769 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:04.313997 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:04.314003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:04.314006 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:00:04.314101 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 09:00:04.314162 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 09:00:04.314890 10612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41349
I20260928 09:00:04.314966 10612 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:04.315138 10612 heartbeater.cc:511] Master 127.0.98.62:41349 requested a full tablet report, sending...
I20260928 09:00:04.315421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10484
I20260928 09:00:04.315524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:00:04.315558 10162 ts_manager.cc:194] Registered new tserver with Master: 5f6573319c5b4eae85457dcb1b70bf90 (127.0.98.3:43221)
I20260928 09:00:04.315732 392 external_mini_cluster.cc:1428] Setting key 710d4944fb6a1651062d998324f8f2e9
I20260928 09:00:04.316605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 09:00:04.318550 10202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:00:04.318059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.0.98.62:41349)
0928 09:00:04.318139 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:04.318142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:04.318240 (+ 98us) server_negotiation.cc:407] Connection header received
0928 09:00:04.318293 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:04.318294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:04.318299 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:04.318348 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:04.318485 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40018: BlockingRecv error: recv got EOF from 127.0.0.1:40018 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 09:00:04.318629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10222
I20260928 09:00:04.323472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10353
I20260928 09:00:04.327907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10484
I20260928 09:00:04.332093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10131
2026-09-28T09:00:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (547 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:00:04 dist-test-slave-286n krb5kdc[10622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:00:04 dist-test-slave-286n krb5kdc[10622](info): set up 2 sockets
Sep 28 09:00:04 dist-test-slave-286n krb5kdc[10622](info): commencing operation
krb5kdc: starting...
W20260928 09:00:05.381018 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:00:05.504251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:05.584933 10635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:05.585139 10635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:05.585161 10635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:05.586357 10635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:00:05.586403 10635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:05.586419 10635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:05.586432 10635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:00:05.586500 10635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:00:05.587978 10635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:05.588240 10635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:05.588490 10635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:05.593515 10641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:05.593535 10643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:05.593535 10640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:05.593890 10635 server_base.cc:1034] running on GCE node
I20260928 09:00:05.594175 10635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:05.594471 10635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:05.595645 10635 hybrid_clock.cc:648] HybridClock initialized: now 1790586005595642 us; error 38 us; skew 500 ppm
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:05.598098 10635 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:00:05.598601 10635 webserver.cc:460] Webserver started at http://127.0.98.62:32941/ using document root <none> and password file <none>
I20260928 09:00:05.598845 10635 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:05.598917 10635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:05.599079 10635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:05.600132 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "b1dc590a737d43e49c07f012c8f96d69"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "34abac5bd014738375faca29688e1ecc"
server_key_iv: "876cd1e45a2a7ca0de5746f292c5bd71"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.600502 10635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1dc590a737d43e49c07f012c8f96d69"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "34abac5bd014738375faca29688e1ecc"
server_key_iv: "876cd1e45a2a7ca0de5746f292c5bd71"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.601869 10635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:05.602602 10650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.602775 10635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.602877 10635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "b1dc590a737d43e49c07f012c8f96d69"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "34abac5bd014738375faca29688e1ecc"
server_key_iv: "876cd1e45a2a7ca0de5746f292c5bd71"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.602963 10635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:05.620353 10635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:05.622551 10635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:05.622731 10635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:05.627002 10635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43927
I20260928 09:00:05.627041 10702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43927 every 8 connection(s)
I20260928 09:00:05.627403 10635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:00:05.628397 10703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:05.629323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10635
I20260928 09:00:05.629491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:00:05.629731 392 external_mini_cluster.cc:1428] Setting key 1e818671fa3e59a95fd0e00342a434e6
I20260928 09:00:05.631866 10703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: Bootstrap starting.
I20260928 09:00:05.632562 10703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:05.632979 10703 log.cc:824] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:05.633826 10703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: No bootstrap required, opened a new log
I20260928 09:00:05.636756 10703 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc590a737d43e49c07f012c8f96d69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43927 } }
I20260928 09:00:05.636910 10703 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:05.636936 10703 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc590a737d43e49c07f012c8f96d69, State: Initialized, Role: FOLLOWER
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:05.637080 10703 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc590a737d43e49c07f012c8f96d69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43927 } }
I20260928 09:00:05.637173 10703 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:00:05.637217 10703 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:00:05.637262 10703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:05.638002 10703 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc590a737d43e49c07f012c8f96d69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43927 } }
I20260928 09:00:05.638108 10703 leader_election.cc:302] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1dc590a737d43e49c07f012c8f96d69; no voters:
I20260928 09:00:05.638343 10703 leader_election.cc:288] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:00:05.638382 10708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:00:05.638492 10708 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [term 1 LEADER]: Becoming Leader. State: Replica: b1dc590a737d43e49c07f012c8f96d69, State: Running, Role: LEADER
I20260928 09:00:05.638579 10708 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc590a737d43e49c07f012c8f96d69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43927 } }
I20260928 09:00:05.638625 10703 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:00:05.640230 10709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1dc590a737d43e49c07f012c8f96d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc590a737d43e49c07f012c8f96d69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43927 } } }
I20260928 09:00:05.640316 10709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:05.640614 10710 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1dc590a737d43e49c07f012c8f96d69. Latest consensus state: current_term: 1 leader_uuid: "b1dc590a737d43e49c07f012c8f96d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1dc590a737d43e49c07f012c8f96d69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43927 } } }
I20260928 09:00:05.640717 10710 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69 [sys.catalog]: This master's current role is: LEADER
W20260928 09:00:05.640954 10722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:00:05.641018 10722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:00:05.641314 10723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:00:05.641549 10723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:00:05.641848 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:05.630296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.0.98.62:43927)
0928 09:00:05.630493 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:05.630500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:05.630526 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:00:05.630909 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:05.630914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:05.630994 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:05.631113 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:05.632010 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.633002 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.634121 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.634552 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.634636 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:05.637647 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:05.637651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:05.637652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:05.637664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:05.639752 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:05.640262 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:05.640264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:05.640265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:05.640373 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:05.640620 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:05.640621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:05.640622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:05.641029 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:05.641181 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:05.641484 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 09:00:05.641646 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260928 09:00:05.645675 10723 catalog_manager.cc:1347] Generated new cluster ID: cfb14a4adc9e40bf85c019f03b3408ca
I20260928 09:00:05.645717 10723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:00:05.662688 10723 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:00:05.663256 10723 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:00:05.665729 10723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: Generated new TSK 0
I20260928 09:00:05.665913 10723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:00:05.703577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43927
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:05.788113 10729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:05.788327 10729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:05.788357 10729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:05.789460 10729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:05.789494 10729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:05.789520 10729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:00:05.790866 10729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:05.791049 10729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:05.791206 10729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:05.796329 10737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:05.796391 10734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:05.796329 10735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:05.796612 10729 server_base.cc:1034] running on GCE node
I20260928 09:00:05.796833 10729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:05.797079 10729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:05.798249 10729 hybrid_clock.cc:648] HybridClock initialized: now 1790586005798248 us; error 50 us; skew 500 ppm
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:05.800913 10729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:00:05.801360 10729 webserver.cc:460] Webserver started at http://127.0.98.1:36963/ using document root <none> and password file <none>
I20260928 09:00:05.801661 10729 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:05.801738 10729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:05.801903 10729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:05.802973 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8100d37947444c2a84e6871b028a6cc"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "96d31e0d7c54084a32e012fa43965aac"
server_key_iv: "db8fe3736f6e4bde85351a40d76d5215"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.803371 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8100d37947444c2a84e6871b028a6cc"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "96d31e0d7c54084a32e012fa43965aac"
server_key_iv: "db8fe3736f6e4bde85351a40d76d5215"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.804783 10729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:00:05.805653 10744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.805848 10729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:00:05.805956 10729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "a8100d37947444c2a84e6871b028a6cc"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "96d31e0d7c54084a32e012fa43965aac"
server_key_iv: "db8fe3736f6e4bde85351a40d76d5215"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.806051 10729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:05.826097 10729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:05.828069 10729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:05.828258 10729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:05.828524 10729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:05.828892 10729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:05.828958 10729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.829000 10729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:05.829023 10729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.836361 10729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35329
I20260928 09:00:05.836459 10857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35329 every 8 connection(s)
I20260928 09:00:05.836848 10729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:00:05.839006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10729
I20260928 09:00:05.839131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:00:05.839332 392 external_mini_cluster.cc:1428] Setting key bcf93427567e226018ca38d069bc7086
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 09:00:05.847012 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:05.837680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50817 (local address 127.0.98.62:43927)
0928 09:00:05.837762 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:05.837764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:05.837872 (+ 108us) server_negotiation.cc:407] Connection header received
0928 09:00:05.838403 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:05.838404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:05.838414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:05.838468 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:05.839734 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.840283 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.841377 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.841585 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.841658 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:05.844033 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:05.844035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:05.844035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:05.844045 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:05.845585 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:05.846024 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:05.846026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:05.846027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:05.846086 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:05.846343 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:05.846344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:05.846344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:05.846460 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:05.846537 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:05.846823 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 09:00:05.846909 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 09:00:05.847203 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:05.837605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43927 (local address 127.0.98.1:50817)
0928 09:00:05.837764 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:05.837801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:05.838313 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:05.838523 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:05.838526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:05.839316 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:05.839621 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:05.839629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.840391 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.840392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:05.841278 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:05.841281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.841663 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.841664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:05.841781 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:05.842327 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:05.842338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:05.843916 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:05.845688 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:05.845695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:05.845697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:05.845925 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:05.846156 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:05.846157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:05.846157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:05.846271 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:05.846540 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:05.846547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:05.846688 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 09:00:05.846792 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 09:00:05.846951 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 09:00:05.847718 10858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43927
I20260928 09:00:05.847805 10858 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:05.847992 10858 heartbeater.cc:511] Master 127.0.98.62:43927 requested a full tablet report, sending...
I20260928 09:00:05.848618 10667 ts_manager.cc:194] Registered new tserver with Master: a8100d37947444c2a84e6871b028a6cc (127.0.98.1:35329)
I20260928 09:00:05.849300 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:00:05.865589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43927
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:05.948067 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:05.948273 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:05.948298 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:05.949422 10863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:05.949465 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:05.949492 10863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:00:05.950951 10863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:05.951198 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:05.951468 10863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:05.956473 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:05.956561 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:05.956705 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:05.956748 10863 server_base.cc:1034] running on GCE node
I20260928 09:00:05.956965 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:05.957214 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:05.958384 10863 hybrid_clock.cc:648] HybridClock initialized: now 1790586005958373 us; error 32 us; skew 500 ppm
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:05.960886 10863 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:00:05.961352 10863 webserver.cc:460] Webserver started at http://127.0.98.2:44665/ using document root <none> and password file <none>
I20260928 09:00:05.961575 10863 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:05.961640 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:05.961796 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:05.962813 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "54f66b4bbcf745f7954e0b2fa26c3cae"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "419c5fc4deac27c7c831d7bda255edc2"
server_key_iv: "a045776f61dc420bdbe9d46201487adc"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.963188 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54f66b4bbcf745f7954e0b2fa26c3cae"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "419c5fc4deac27c7c831d7bda255edc2"
server_key_iv: "a045776f61dc420bdbe9d46201487adc"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.964435 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:00:05.965227 10878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.965442 10863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.965538 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "54f66b4bbcf745f7954e0b2fa26c3cae"
format_stamp: "Formatted at 2026-09-28 09:00:05 on dist-test-slave-286n"
server_key: "419c5fc4deac27c7c831d7bda255edc2"
server_key_iv: "a045776f61dc420bdbe9d46201487adc"
server_key_version: "encryptionkey@0"
I20260928 09:00:05.965631 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:05.975993 10863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:05.977828 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:05.977985 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:05.978197 10863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:05.978477 10863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:05.978528 10863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.978574 10863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:05.978621 10863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:05.985121 10863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37779
I20260928 09:00:05.985169 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37779 every 8 connection(s)
I20260928 09:00:05.985481 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:00:05.990309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10863
I20260928 09:00:05.990414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:00:05.990734 392 external_mini_cluster.cc:1428] Setting key 6bb675eef4860dede21bfd97887fc7e8
Sep 28 09:00:05 dist-test-slave-286n krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:05.995712 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:05.986162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42765 (local address 127.0.98.62:43927)
0928 09:00:05.986224 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:05.986226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:05.986409 (+ 183us) server_negotiation.cc:407] Connection header received
0928 09:00:05.986991 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:05.986992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:05.987005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:05.987068 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:05.988021 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.988526 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.989755 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.990078 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.990173 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:05.992532 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:05.992533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:05.992534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:05.992541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:05.994107 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:05.994609 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:05.994610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:05.994610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:05.994667 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:05.994950 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:05.994951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:05.994951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:05.995052 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:05.995107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:05.995425 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 09:00:05.995492 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:00:05.995882 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:05.986141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43927 (local address 127.0.98.2:42765)
0928 09:00:05.986292 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:05.986326 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:05.986845 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:05.987143 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:05.987147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:05.987704 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:05.987884 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:05.987888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.988622 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.988623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:05.989635 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:05.989637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:05.990202 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:05.990204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:05.990357 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:05.990978 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:05.990986 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:05.992354 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:05.994191 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:05.994198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:05.994201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:05.994513 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:05.994741 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:05.994742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:05.994742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:05.994886 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:05.995116 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:05.995122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:05.995289 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 09:00:05.995407 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 09:00:05.995588 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260928 09:00:05.996472 10992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43927
I20260928 09:00:05.996587 10992 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:05.996825 10992 heartbeater.cc:511] Master 127.0.98.62:43927 requested a full tablet report, sending...
I20260928 09:00:05.997246 10667 ts_manager.cc:194] Registered new tserver with Master: 54f66b4bbcf745f7954e0b2fa26c3cae (127.0.98.2:37779)
I20260928 09:00:05.997632 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42765
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:00:06.019992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43927
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:06.105087 10997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:06.105314 10997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:06.105341 10997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:06.107255 10997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:06.107314 10997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:06.107369 10997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:00:06.108986 10997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:06.109308 10997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:06.109576 10997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:06.114837 11005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:06.114868 11003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:06.114837 11002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:06.115057 10997 server_base.cc:1034] running on GCE node
I20260928 09:00:06.115298 10997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:06.115514 10997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:06.116700 10997 hybrid_clock.cc:648] HybridClock initialized: now 1790586006116686 us; error 48 us; skew 500 ppm
Sep 28 09:00:06 dist-test-slave-286n krb5kdc[10622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:06.119210 10997 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:00:06.119598 10997 webserver.cc:460] Webserver started at http://127.0.98.3:39667/ using document root <none> and password file <none>
I20260928 09:00:06.119788 10997 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:06.119827 10997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:06.119915 10997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:00:06.120990 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac"
format_stamp: "Formatted at 2026-09-28 09:00:06 on dist-test-slave-286n"
server_key: "697072e3b75ec966aa936272043d7269"
server_key_iv: "0a39e000213a8f3d9e4f471d2ec8786c"
server_key_version: "encryptionkey@0"
I20260928 09:00:06.121290 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac"
format_stamp: "Formatted at 2026-09-28 09:00:06 on dist-test-slave-286n"
server_key: "697072e3b75ec966aa936272043d7269"
server_key_iv: "0a39e000213a8f3d9e4f471d2ec8786c"
server_key_version: "encryptionkey@0"
I20260928 09:00:06.122540 10997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:00:06.123240 11012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:06.123435 10997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:06.123518 10997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac"
format_stamp: "Formatted at 2026-09-28 09:00:06 on dist-test-slave-286n"
server_key: "697072e3b75ec966aa936272043d7269"
server_key_iv: "0a39e000213a8f3d9e4f471d2ec8786c"
server_key_version: "encryptionkey@0"
I20260928 09:00:06.123564 10997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:06.150313 10997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:06.152616 10997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:06.152792 10997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:06.153039 10997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:06.153358 10997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:06.153393 10997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:06.153442 10997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:06.153465 10997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:06.159066 10997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34647
I20260928 09:00:06.159152 11125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34647 every 8 connection(s)
I20260928 09:00:06.159443 10997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:00:06.164997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 10997
I20260928 09:00:06.165103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:00:06.165282 392 external_mini_cluster.cc:1428] Setting key 435a58c99d74e34c80b948582e175843
Sep 28 09:00:06 dist-test-slave-286n krb5kdc[10622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:06.168758 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.160200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51667 (local address 127.0.98.62:43927)
0928 09:00:06.160296 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.160298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.160459 (+ 161us) server_negotiation.cc:407] Connection header received
0928 09:00:06.161045 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.161046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.161057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.161120 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:06.162022 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.162530 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.163547 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.163752 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.163823 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.166282 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:06.166283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:06.166284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:06.166291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:06.167427 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:06.167830 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:06.167831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:06.167832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:06.167889 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:06.168159 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:06.168160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:06.168160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:06.168282 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:06.168339 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.168568 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.168629 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 09:00:06.168938 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.160123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43927 (local address 127.0.98.3:51667)
0928 09:00:06.160334 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.160378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:06.160943 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:06.161193 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:06.161196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:06.161742 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:06.161924 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.161928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.162624 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.162625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.163447 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.163449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.163827 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.163828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.163944 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.164444 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:06.164452 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:06.166172 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:06.167529 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:06.167535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:06.167538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:06.167753 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:06.167968 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:06.167968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:06.167969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:06.168073 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:06.168343 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:06.168347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:06.168460 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 09:00:06.168544 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:00:06.168705 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260928 09:00:06.169368 11126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43927
I20260928 09:00:06.169466 11126 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:06.169628 11126 heartbeater.cc:511] Master 127.0.98.62:43927 requested a full tablet report, sending...
I20260928 09:00:06.170033 10667 ts_manager.cc:194] Registered new tserver with Master: 5c6a1c1988e048d3a78f6f6a8f7452ac (127.0.98.3:34647)
I20260928 09:00:06.170388 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51667
I20260928 09:00:06.177145 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:00:06.184302 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.178431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.0.98.62:43927)
0928 09:00:06.178516 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.178518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.178572 (+ 54us) server_negotiation.cc:407] Connection header received
0928 09:00:06.178612 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.178613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.178621 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.178669 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:06.179175 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.179738 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.180487 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.180751 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.180830 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.181898 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:06.181900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:06.181900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:06.181906 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:06.182991 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:06.183345 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:06.183346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:06.183347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:06.183543 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:06.183770 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:06.183771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:06.183771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:06.183917 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:06.184022 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.184124 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.184203 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 09:00:06.186033 10667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:00:06.187561 10667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:00:06.194778 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.190121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.0.98.3:34647)
0928 09:00:06.190408 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.190411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.190760 (+ 349us) server_negotiation.cc:407] Connection header received
0928 09:00:06.190978 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.190982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.190998 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.191066 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:06.191217 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.192036 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.193532 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.194182 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.194284 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.194345 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.194573 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.194650 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":88,"threads_started":1}
I20260928 09:00:06.194943 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.190001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37779 (local address 127.0.0.1:35878)
0928 09:00:06.190550 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.190673 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:06.190784 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:06.191090 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:06.191091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:06.191093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:06.191166 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.191168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.192606 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.192607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.193748 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.193750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.194693 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.194694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.194764 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.194768 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:00:06.194820 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:00:06.194860 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":28,"thread_start_us":60,"threads_started":1}
I20260928 09:00:06.195044 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.190430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.0.98.1:35329)
0928 09:00:06.190610 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.190615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.190971 (+ 356us) server_negotiation.cc:407] Connection header received
0928 09:00:06.191168 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.191171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.191187 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.191304 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:06.191426 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.192744 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.193906 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.194589 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.194645 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.194694 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.194901 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.194941 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260928 09:00:06.195250 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.190158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.0.98.2:37779)
0928 09:00:06.190365 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.190370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.190814 (+ 444us) server_negotiation.cc:407] Connection header received
0928 09:00:06.190896 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.190899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.190916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.190992 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:06.191249 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.192424 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.193825 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.194528 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.194601 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.194658 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.195125 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.195172 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260928 09:00:06.195591 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.190342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35329 (local address 127.0.0.1:33826)
0928 09:00:06.190913 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.190921 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:06.191023 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:06.191273 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:06.191274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:06.191276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:06.191321 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.191322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.192848 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.192849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.193835 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.193837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.194721 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.194722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.194783 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.194787 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:00:06.194831 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 09:00:06.194861 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260928 09:00:06.196204 10926 tablet_service.cc:1467] Processing CreateTablet for tablet 9d81051d68b24436a23febd035d1e0cf (DEFAULT_TABLE table=test-table [id=ed28ef612ac0409ea85c342f034a421f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:00:06.196301 10791 tablet_service.cc:1467] Processing CreateTablet for tablet 9d81051d68b24436a23febd035d1e0cf (DEFAULT_TABLE table=test-table [id=ed28ef612ac0409ea85c342f034a421f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:00:06.196548 10926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d81051d68b24436a23febd035d1e0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:06.196652 10791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d81051d68b24436a23febd035d1e0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:06.200012 11144 tablet_bootstrap.cc:492] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc: Bootstrap starting.
I20260928 09:00:06.200070 11143 tablet_bootstrap.cc:492] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Bootstrap starting.
I20260928 09:00:06.200690 11144 tablet_bootstrap.cc:654] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:06.200837 11143 tablet_bootstrap.cc:654] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:06.201215 11144 log.cc:824] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:06.201452 11143 log.cc:824] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:06.201898 11144 tablet_bootstrap.cc:492] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc: No bootstrap required, opened a new log
I20260928 09:00:06.201956 11144 ts_tablet_manager.cc:1397] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 09:00:06.202160 11143 tablet_bootstrap.cc:492] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: No bootstrap required, opened a new log
I20260928 09:00:06.202209 11143 ts_tablet_manager.cc:1397] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 09:00:06.203724 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.190024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34647 (local address 127.0.0.1:33308)
0928 09:00:06.190696 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.190705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:06.190881 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:06.191075 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:06.191078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:06.191081 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:06.191160 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.191162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.192162 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.192163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.193418 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.193420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.194300 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.194301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.194379 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.194392 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:00:06.194462 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 09:00:06.203651 (+ 9189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":122,"threads_started":1}
I20260928 09:00:06.203742 11144 raft_consensus.cc:348] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.203897 11144 raft_consensus.cc:374] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:06.203925 11144 raft_consensus.cc:729] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8100d37947444c2a84e6871b028a6cc, State: Initialized, Role: FOLLOWER
I20260928 09:00:06.204093 11144 consensus_queue.cc:260] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.204183 11147 raft_consensus.cc:482] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:00:06.204052 11143 raft_consensus.cc:348] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.204258 11143 raft_consensus.cc:374] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:06.204293 11143 raft_consensus.cc:729] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f66b4bbcf745f7954e0b2fa26c3cae, State: Initialized, Role: FOLLOWER
I20260928 09:00:06.204273 11147 raft_consensus.cc:504] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.204325 10858 heartbeater.cc:503] Master 127.0.98.62:43927 was elected leader, sending a full tablet report...
I20260928 09:00:06.204399 11143 consensus_queue.cc:260] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.204605 11147 leader_election.cc:288] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6a1c1988e048d3a78f6f6a8f7452ac (127.0.98.3:34647), 54f66b4bbcf745f7954e0b2fa26c3cae (127.0.98.2:37779)
I20260928 09:00:06.204658 11143 ts_tablet_manager.cc:1428] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 09:00:06.204761 10992 heartbeater.cc:503] Master 127.0.98.62:43927 was elected leader, sending a full tablet report...
I20260928 09:00:06.204975 11060 tablet_service.cc:1467] Processing CreateTablet for tablet 9d81051d68b24436a23febd035d1e0cf (DEFAULT_TABLE table=test-table [id=ed28ef612ac0409ea85c342f034a421f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:00:06.205278 11060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d81051d68b24436a23febd035d1e0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:06.204325 11144 ts_tablet_manager.cc:1428] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 09:00:06.208753 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.205080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37779 (local address 127.0.98.1:38347)
0928 09:00:06.205443 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.205455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:06.205502 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:06.205763 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:06.205764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:06.205766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:06.205827 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.205829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.206798 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.206799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.207540 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.207541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.208496 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.208496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.208572 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.208575 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:00:06.208606 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 09:00:06.208664 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260928 09:00:06.208791 11151 tablet_bootstrap.cc:492] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Bootstrap starting.
I20260928 09:00:06.208963 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.205124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38347 (local address 127.0.98.2:37779)
0928 09:00:06.205158 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.205160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.205637 (+ 477us) server_negotiation.cc:407] Connection header received
0928 09:00:06.205661 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.205662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.205672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.205721 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:06.206077 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.206694 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.207626 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.208393 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.208850 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.208860 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.208877 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.208910 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:00:06.209161 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d81051d68b24436a23febd035d1e0cf" candidate_uuid: "a8100d37947444c2a84e6871b028a6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" is_pre_election: true
I20260928 09:00:06.209270 10946 raft_consensus.cc:2454] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8100d37947444c2a84e6871b028a6cc in term 0.
I20260928 09:00:06.209545 10748 leader_election.cc:302] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f66b4bbcf745f7954e0b2fa26c3cae, a8100d37947444c2a84e6871b028a6cc; no voters:
I20260928 09:00:06.209570 11151 tablet_bootstrap.cc:654] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:06.209722 11147 raft_consensus.cc:2781] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:00:06.209833 11147 raft_consensus.cc:482] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:00:06.209908 11147 raft_consensus.cc:3037] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:06.210103 11151 log.cc:824] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:06.210788 11151 tablet_bootstrap.cc:492] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: No bootstrap required, opened a new log
I20260928 09:00:06.210747 11147 raft_consensus.cc:504] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.210853 11151 ts_tablet_manager.cc:1397] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:00:06.211764 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.208008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35503 (local address 127.0.98.3:34647)
0928 09:00:06.208048 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.208050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.208128 (+ 78us) server_negotiation.cc:407] Connection header received
0928 09:00:06.208268 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.208269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.208279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.208306 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:06.208501 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.209191 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.210813 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.211422 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.211467 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.211476 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.211672 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.211705 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:00:06.212088 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.207957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34647 (local address 127.0.98.1:35503)
0928 09:00:06.208077 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.208087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:06.208183 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:06.208361 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:06.208361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:06.208363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:06.208423 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.208425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.209564 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.209564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.210708 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:06.210711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.211533 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.211534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:06.211602 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.211605 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:00:06.211859 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 09:00:06.212012 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260928 09:00:06.212383 11080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d81051d68b24436a23febd035d1e0cf" candidate_uuid: "a8100d37947444c2a84e6871b028a6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" is_pre_election: true
I20260928 09:00:06.212617 11079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d81051d68b24436a23febd035d1e0cf" candidate_uuid: "a8100d37947444c2a84e6871b028a6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac"
W20260928 09:00:06.212894 10746 leader_election.cc:341] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5c6a1c1988e048d3a78f6f6a8f7452ac (127.0.98.3:34647): Illegal state: must be running to vote when last-logged opid is not known
W20260928 09:00:06.213009 10746 leader_election.cc:341] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5c6a1c1988e048d3a78f6f6a8f7452ac (127.0.98.3:34647): Illegal state: must be running to vote when last-logged opid is not known
I20260928 09:00:06.213109 11147 leader_election.cc:288] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [CANDIDATE]: Term 1 election: Requested vote from peers 5c6a1c1988e048d3a78f6f6a8f7452ac (127.0.98.3:34647), 54f66b4bbcf745f7954e0b2fa26c3cae (127.0.98.2:37779)
I20260928 09:00:06.213239 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d81051d68b24436a23febd035d1e0cf" candidate_uuid: "a8100d37947444c2a84e6871b028a6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae"
I20260928 09:00:06.213327 10946 raft_consensus.cc:3037] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:06.213315 11151 raft_consensus.cc:348] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.213449 11151 raft_consensus.cc:374] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:06.213505 11151 raft_consensus.cc:729] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6a1c1988e048d3a78f6f6a8f7452ac, State: Initialized, Role: FOLLOWER
I20260928 09:00:06.213629 11151 consensus_queue.cc:260] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.213915 11151 ts_tablet_manager.cc:1428] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:00:06.213932 11126 heartbeater.cc:503] Master 127.0.98.62:43927 was elected leader, sending a full tablet report...
I20260928 09:00:06.214216 10946 raft_consensus.cc:2454] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8100d37947444c2a84e6871b028a6cc in term 1.
I20260928 09:00:06.214370 10748 leader_election.cc:302] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54f66b4bbcf745f7954e0b2fa26c3cae, a8100d37947444c2a84e6871b028a6cc; no voters: 5c6a1c1988e048d3a78f6f6a8f7452ac
I20260928 09:00:06.214462 11147 raft_consensus.cc:2781] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:00:06.214535 11147 raft_consensus.cc:686] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [term 1 LEADER]: Becoming Leader. State: Replica: a8100d37947444c2a84e6871b028a6cc, State: Running, Role: LEADER
I20260928 09:00:06.214615 11147 consensus_queue.cc:237] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } }
I20260928 09:00:06.215408 10667 catalog_manager.cc:5495] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc reported cstate change: term changed from 0 to 1, leader changed from <none> to a8100d37947444c2a84e6871b028a6cc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a8100d37947444c2a84e6871b028a6cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8100d37947444c2a84e6871b028a6cc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 } health_report { overall_health: UNKNOWN } } }
I20260928 09:00:06.222762 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.219376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.0.98.1:35329)
0928 09:00:06.219419 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:06.219420 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:06.219430 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:00:06.219594 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:06.219595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:06.219617 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:06.219684 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:00:06.219858 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.220590 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.221808 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:06.222106 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:06.222190 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:06.222370 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:00:06.222499 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:00:06.222580 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:06.222642 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 09:00:06.222662 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:00:06.224488 10946 raft_consensus.cc:1261] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 1 FOLLOWER]: Refusing update from remote peer a8100d37947444c2a84e6871b028a6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:00:06.224450 11079 raft_consensus.cc:3037] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:06.224860 11148 consensus_queue.cc:1035] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f66b4bbcf745f7954e0b2fa26c3cae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:00:06.225519 11079 raft_consensus.cc:1261] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 1 FOLLOWER]: Refusing update from remote peer a8100d37947444c2a84e6871b028a6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:00:06.225760 11148 consensus_queue.cc:1035] T 9d81051d68b24436a23febd035d1e0cf P a8100d37947444c2a84e6871b028a6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6a1c1988e048d3a78f6f6a8f7452ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:00:06.229446 10667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:00:06.229553 10667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:00:06.230340 10667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1dc590a737d43e49c07f012c8f96d69: Sending DeleteTablet for 3 replicas of tablet 9d81051d68b24436a23febd035d1e0cf
I20260928 09:00:06.230705 11060 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d81051d68b24436a23febd035d1e0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:00:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260928 09:00:06.230911 11165 tablet_replica.cc:330] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: stopping tablet replica
I20260928 09:00:06.230926 10926 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d81051d68b24436a23febd035d1e0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:00:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35878
I20260928 09:00:06.231014 11165 raft_consensus.cc:2229] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:00:06.231109 10791 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d81051d68b24436a23febd035d1e0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:00:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33826
I20260928 09:00:06.231137 11165 raft_consensus.cc:2258] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:00:06.231528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10729
I20260928 09:00:06.232848 11165 ts_tablet_manager.cc:1905] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:00:06.233652 11166 tablet_replica.cc:330] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: stopping tablet replica
I20260928 09:00:06.233788 11166 raft_consensus.cc:2229] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:00:06.233902 11166 raft_consensus.cc:2258] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:00:06.234381 11166 ts_tablet_manager.cc:1905] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:00:06.234943 11165 ts_tablet_manager.cc:1918] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:00:06.235040 11165 log.cc:1196] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/9d81051d68b24436a23febd035d1e0cf
I20260928 09:00:06.235306 11165 ts_tablet_manager.cc:1939] T 9d81051d68b24436a23febd035d1e0cf P 5c6a1c1988e048d3a78f6f6a8f7452ac: Deleting consensus metadata
I20260928 09:00:06.235699 10652 catalog_manager.cc:4841] TS 5c6a1c1988e048d3a78f6f6a8f7452ac (127.0.98.3:34647): tablet 9d81051d68b24436a23febd035d1e0cf (table test-table [id=ed28ef612ac0409ea85c342f034a421f]) successfully deleted
I20260928 09:00:06.236457 11166 ts_tablet_manager.cc:1918] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:00:06.236541 11166 log.cc:1196] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/9d81051d68b24436a23febd035d1e0cf
I20260928 09:00:06.236881 11166 ts_tablet_manager.cc:1939] T 9d81051d68b24436a23febd035d1e0cf P 54f66b4bbcf745f7954e0b2fa26c3cae: Deleting consensus metadata
I20260928 09:00:06.237315 10654 catalog_manager.cc:4841] TS 54f66b4bbcf745f7954e0b2fa26c3cae (127.0.98.2:37779): tablet 9d81051d68b24436a23febd035d1e0cf (table test-table [id=ed28ef612ac0409ea85c342f034a421f]) successfully deleted
W20260928 09:00:06.238081 10652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35329 (error 108)
I20260928 09:00:06.238328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10863
I20260928 09:00:06.239503 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:06.239317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35329 (local address 127.0.0.1:33834)
0928 09:00:06.239397 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:06.239478 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 09:00:06.239701 10652 catalog_manager.cc:4584] TS a8100d37947444c2a84e6871b028a6cc (127.0.98.1:35329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d81051d68b24436a23febd035d1e0cf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35329: connect: Connection refused (error 111)
I20260928 09:00:06.243728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10997
I20260928 09:00:06.248184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 10635
2026-09-28T09:00:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1916 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:00:06 dist-test-slave-286n krb5kdc[11171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:00:06 dist-test-slave-286n krb5kdc[11171](info): set up 2 sockets
Sep 28 09:00:06 dist-test-slave-286n krb5kdc[11171](info): commencing operation
krb5kdc: starting...
W20260928 09:00:07.289337 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:00:07 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:00:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 09:00:07.464216 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaJshIb/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/postgres -l logfile start
2026-09-28 09:00:10.160 UTC [11201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 09:00:10.160 UTC [11201] LOG: listening on IPv4 address "127.0.98.20", port 54077
2026-09-28 09:00:10.163 UTC [11201] LOG: listening on Unix socket "/tmp/.s.PGSQL.54077"
2026-09-28 09:00:10.186 UTC [11204] LOG: database system was shut down at 2026-09-28 09:00:08 UTC
2026-09-28 09:00:10.190 UTC [11201] LOG: database system is ready to accept connections
I20260928 09:00:11.144680 392 mini_postgres.cc:96] Postgres bound to 54077
2026-09-28 09:00:11.150 UTC [11212] FATAL: database "slave" does not exist
127.0.98.20:54077 - accepting connections
I20260928 09:00:11.151293 392 mini_ranger.cc:161] Starting Ranger...
I20260928 09:00:11.166473 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 09:00:11.758351 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 09:00:11.758457 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-admin
2026-09-28 09:00:12,077 [I] DB FLAVOR :POSTGRES
2026-09-28 09:00:12,078 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:00:12,078 [I] Checking connection..
2026-09-28 09:00:12,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 09:00:12,343 [I] Checking connection passed.
2026-09-28 09:00:12,343 [I] --------- Verifying version history table ---------
2026-09-28 09:00:12,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:00:12,610 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:00:12,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:00:12,871 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:00:12,872 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 09:00:12,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 09:00:13.110 UTC [11276] WARNING: there is no transaction in progress
2026-09-28 09:00:13,117 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 09:00:13,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:00:13,374 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 09:00:13,374 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:00:13,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 09:00:13,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 09:00:13,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-286n.c.gcp-upstream.internal','N') ;"
2026-09-28 09:00:14,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:00:14,376 [I] Table x_portal_user does not exist in database ranger
2026-09-28 09:00:14,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:00:14,629 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 09:00:14,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:00:14,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:00:15,121 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 09:00:15,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 09:00:16.176 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.195 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.214 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.230 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.255 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.276 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.339 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.347 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.363 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.376 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.388 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.404 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.414 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.422 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.430 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:16.437 UTC [11384] WARNING: there is no transaction in progress
2026-09-28 09:00:17,134 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 09:00:17,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 09:00:17,375 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 09:00:17,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:00:17,611 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 09:00:17,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:00:17,883 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 09:00:17,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:00:18,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:00:18,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-286n.c.gcp-upstream.internal';"
2026-09-28 09:00:18,645 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 09:00:18,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 09:00:18,898 [I] DB_PATCHES have already been applied
I20260928 09:00:18.904253 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaJshIb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 09:00:18.904322 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 09:00:18.907383 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58743
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:00:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 09:00:20.003381 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.852s user 0.001s sys 0.008s
Sep 28, 2026 9:00:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:00:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:00:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:00:40 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:00:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 09:00:42.319362 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 09:00:42 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586042, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 09:00:42.404944 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 09:00:42.414143 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 09:00:43.310439 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 09:00:43.310535 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms
I20260928 09:00:43.310554 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54077
2026-09-28 09:00:43,486 [I] DB FLAVOR :POSTGRES
2026-09-28 09:00:43,486 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:00:43,486 [I] Checking connection
2026-09-28 09:00:43,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 09:00:43,746 [I] connection success
2026-09-28 09:00:43,747 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 09:00:43,747 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 09:00:43,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 09:00:44,012 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 09:00:44,012 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:00:44,012 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 09:00:44,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 09:00:44,276 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 09:00:44.777704 392 mini_ranger_kms.cc:310] Created kms service
I20260928 09:00:44.978503 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 09:00:45.024093 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 09:00:45 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586042, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:00:45.409171 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 09:00:45.409452 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 09:00:45.409487 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 09:00:45.412549 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11564
I20260928 09:00:45.412699 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36663
Sep 28, 2026 9:00:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:00:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:00:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:00:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:00:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:00:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 09:00:47.641075 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.236s user 0.005s sys 0.152s
I20260928 09:00:47.641175 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 09:00:47.641243 392 mini_ranger_kms.cc:401] 127.0.98.20:36663/kms/v1/keys
Sep 28 09:00:47 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586047, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 09:00:48 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586047, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:00:48 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586047, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:00:50 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:00:51.033955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37765
--ranger_config_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36663/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:51.119722 11615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:51.119959 11615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:51.119997 11615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:51.121222 11615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:00:51.121286 11615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:51.121313 11615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:51.121343 11615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:00:51.121369 11615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:00:51.122905 11615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36663/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37765
--ranger_config_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:51.123186 11615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:51.123409 11615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:51.128508 11620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:51.128542 11621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:51.128805 11615 server_base.cc:1034] running on GCE node
W20260928 09:00:51.128626 11623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:51.129113 11615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:51.129433 11615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:51.130599 11615 hybrid_clock.cc:648] HybridClock initialized: now 1790586051130587 us; error 69 us; skew 500 ppm
Sep 28 09:00:51 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:51.133047 11615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:00:51.133500 11615 webserver.cc:460] Webserver started at http://127.0.98.62:40997/ using document root <none> and password file <none>
I20260928 09:00:51.133718 11615 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:51.133777 11615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:51.133931 11615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:00:51 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:00:51.186568 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "a95b6b7f07b64f40bb3916680fa15c6d"
format_stamp: "Formatted at 2026-09-28 09:00:51 on dist-test-slave-286n"
server_key: "4f31010db0dd9bdaf12728a67ba64875"
server_key_iv: "8acd32ccee62543cc5f566fa8241d900"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:51.187052 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "a95b6b7f07b64f40bb3916680fa15c6d"
format_stamp: "Formatted at 2026-09-28 09:00:51 on dist-test-slave-286n"
server_key: "4f31010db0dd9bdaf12728a67ba64875"
server_key_iv: "8acd32ccee62543cc5f566fa8241d900"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:51.189004 11615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 09:00:51.204484 11630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:51.204878 11615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:51.204982 11615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "a95b6b7f07b64f40bb3916680fa15c6d"
format_stamp: "Formatted at 2026-09-28 09:00:51 on dist-test-slave-286n"
server_key: "4f31010db0dd9bdaf12728a67ba64875"
server_key_iv: "8acd32ccee62543cc5f566fa8241d900"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:51.205094 11615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:51.215603 11615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:51.217507 11615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:51.217661 11615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:51.222491 11615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37765
I20260928 09:00:51.222553 11682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37765 every 8 connection(s)
I20260928 09:00:51.222899 11615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:00:51.223630 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:51.227240 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d: Bootstrap starting.
I20260928 09:00:51.227942 11683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:51.228216 11683 log.cc:824] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:51.229053 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d: No bootstrap required, opened a new log
I20260928 09:00:51.230007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 11615
I20260928 09:00:51.230137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:00:51.230336 392 external_mini_cluster.cc:1428] Setting key 651b2b279af7b1f0db0d028c518c625f
I20260928 09:00:51.231805 11683 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260928 09:00:51.231979 11683 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:51.232012 11683 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95b6b7f07b64f40bb3916680fa15c6d, State: Initialized, Role: FOLLOWER
I20260928 09:00:51.232155 11683 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260928 09:00:51.232228 11683 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:00:51.232259 11683 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:00:51.232329 11683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:51.233011 11683 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260928 09:00:51.233117 11683 leader_election.cc:302] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95b6b7f07b64f40bb3916680fa15c6d; no voters:
I20260928 09:00:51.233305 11683 leader_election.cc:288] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:00:51.233359 11688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:00:51.233608 11683 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:00:51.234752 11688 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [term 1 LEADER]: Becoming Leader. State: Replica: a95b6b7f07b64f40bb3916680fa15c6d, State: Running, Role: LEADER
I20260928 09:00:51.234947 11688 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } }
I20260928 09:00:51.235323 11688 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95b6b7f07b64f40bb3916680fa15c6d. Latest consensus state: current_term: 1 leader_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } } }
I20260928 09:00:51.236251 11683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 09:00:51.236545 11689 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95b6b7f07b64f40bb3916680fa15c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37765 } } }
I20260928 09:00:51.236727 11689 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:51.236627 11688 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d [sys.catalog]: This master's current role is: LEADER
I20260928 09:00:51.237561 11702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:00:51.238042 11702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:00:51 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:51.242055 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:51.230879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.98.62:37765)
0928 09:00:51.231066 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:51.231075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:51.232284 (+ 1209us) server_negotiation.cc:407] Connection header received
0928 09:00:51.232511 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:51.232517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:51.232566 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:51.232760 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:51.233481 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:51.234525 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:51.235604 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:51.236175 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:51.236303 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:51.238884 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:51.238888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:51.238888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:51.238900 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:51.240494 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:51.240931 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:51.240934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:51.240934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:51.241016 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:51.241223 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:51.241224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:51.241224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:51.241455 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:51.241613 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:51.241722 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 09:00:51.241836 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260928 09:00:51.246454 11702 catalog_manager.cc:1347] Generated new cluster ID: b58a70ab27ba492b9cabaeb46ed5a529
I20260928 09:00:51.246624 11702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:00:51.268707 11702 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:00:51.269268 11702 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:00:51.273941 11702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d: Generated new TSK 0
I20260928 09:00:51.274118 11702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:00:51.992559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37765
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:52.107146 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:52.107509 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:52.107585 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:52.109457 11728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:52.109571 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:52.109633 11728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:00:52.112069 11728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:52.112344 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:52.112711 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:52.120232 11733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:52.120229 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:52.120482 11734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:52.120913 11728 server_base.cc:1034] running on GCE node
I20260928 09:00:52.121208 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:52.121513 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:52.122761 11728 hybrid_clock.cc:648] HybridClock initialized: now 1790586052122743 us; error 35 us; skew 500 ppm
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:52.125774 11728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:00:52.126387 11728 webserver.cc:460] Webserver started at http://127.0.98.1:39517/ using document root <none> and password file <none>
I20260928 09:00:52.126677 11728 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:52.126737 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:52.126937 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:00:52.147774 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "da368a08ea744c379e2925072a866b5b"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "be5b193020650f7a4e1f6f817f4f4797"
server_key_iv: "fef621b062a9349bbd7f5c1278df4ac8"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.148279 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da368a08ea744c379e2925072a866b5b"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "be5b193020650f7a4e1f6f817f4f4797"
server_key_iv: "fef621b062a9349bbd7f5c1278df4ac8"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.150467 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 09:00:52.176769 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.178712 11728 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260928 09:00:52.178850 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "da368a08ea744c379e2925072a866b5b"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "be5b193020650f7a4e1f6f817f4f4797"
server_key_iv: "fef621b062a9349bbd7f5c1278df4ac8"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.178978 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:52.199625 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:52.202600 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:52.202859 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:52.203218 11728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:52.203682 11728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:52.203794 11728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.203893 11728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:52.203972 11728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.211990 11728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38637
I20260928 09:00:52.212531 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:00:52.215298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 11728
I20260928 09:00:52.215407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:00:52.215629 392 external_mini_cluster.cc:1428] Setting key 9471331a0a4f2550643545ab55656dbd
I20260928 09:00:52.218606 11858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38637 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:52.239238 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.219333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37765 (local address 127.0.98.1:57335)
0928 09:00:52.219640 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.219685 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.220361 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.220891 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.220896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.221191 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:52.221503 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.221510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.222757 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.222758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.224321 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.224324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.232070 (+ 7746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.232072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.232267 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.233043 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:52.233055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:52.234964 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:52.237366 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:52.237374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:52.237377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:52.237691 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:52.237942 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:52.237942 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:52.237943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:52.238096 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:52.238428 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:52.238435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:52.238648 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.238786 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.238965 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
I20260928 09:00:52.239876 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.219400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57335 (local address 127.0.98.62:37765)
0928 09:00:52.220579 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.220581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.220590 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:00:52.220617 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.220618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.220631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.220746 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:52.221632 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.222629 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.231462 (+ 8833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.231869 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.231970 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.235090 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:52.235092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:52.235093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:52.235101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:52.237261 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.237800 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.237802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.237802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.237870 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.238174 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.238175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.238176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.238334 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:52.238389 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.239651 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.239756 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":71,"threads_started":1}
I20260928 09:00:52.240686 11859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37765
I20260928 09:00:52.240798 11859 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:52.240998 11859 heartbeater.cc:511] Master 127.0.98.62:37765 requested a full tablet report, sending...
I20260928 09:00:52.241662 11647 ts_manager.cc:194] Registered new tserver with Master: da368a08ea744c379e2925072a866b5b (127.0.98.1:38637)
I20260928 09:00:52.242673 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:00:52.275988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37765
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 09:00:52.415271 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:52.415575 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:52.415611 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:52.417508 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:52.417562 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:52.417603 11867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:00:52.419950 11867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:52.420210 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:52.420490 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:52.426267 11872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:52.426275 11875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:52.426273 11873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:52.426651 11867 server_base.cc:1034] running on GCE node
I20260928 09:00:52.426862 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:52.427101 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:52.428265 11867 hybrid_clock.cc:648] HybridClock initialized: now 1790586052428254 us; error 33 us; skew 500 ppm
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:52.430912 11867 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:00:52.431346 11867 webserver.cc:460] Webserver started at http://127.0.98.2:35363/ using document root <none> and password file <none>
I20260928 09:00:52.431612 11867 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:52.431687 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:52.431844 11867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:00:52.454391 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "bccf1bfa73e44d73882075146ac6d631"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "2f532c3ca255277b2a20728728b80d1e"
server_key_iv: "3ecb975ad528460ce05a6740c3a6cfda"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.454860 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bccf1bfa73e44d73882075146ac6d631"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "2f532c3ca255277b2a20728728b80d1e"
server_key_iv: "3ecb975ad528460ce05a6740c3a6cfda"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.456696 11867 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 09:00:52.476814 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.477102 11867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 09:00:52.477253 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "bccf1bfa73e44d73882075146ac6d631"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "2f532c3ca255277b2a20728728b80d1e"
server_key_iv: "3ecb975ad528460ce05a6740c3a6cfda"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.477442 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:52.497273 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:52.500092 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:52.500319 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:52.500584 11867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:52.501085 11867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:52.501187 11867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.501266 11867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:52.501328 11867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.508948 11867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46877
I20260928 09:00:52.509509 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:00:52.513018 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46877 every 8 connection(s)
I20260928 09:00:52.514055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 11867
I20260928 09:00:52.514165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:00:52.514386 392 external_mini_cluster.cc:1428] Setting key 05790616887f0d51000a58ad02922734
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:52.522562 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.510910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44185 (local address 127.0.98.62:37765)
0928 09:00:52.510959 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.510961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.510973 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:00:52.511883 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.511885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.511900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.511939 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:52.512965 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.513624 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.515244 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.515612 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.515662 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.518725 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:52.518758 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:52.518759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:52.518767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:52.520804 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.521337 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.521338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.521338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.521405 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.521720 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.521721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.521721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.521879 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:52.521926 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.522350 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.522453 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:00:52.523398 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.510603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37765 (local address 127.0.98.2:44185)
0928 09:00:52.510783 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.510831 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.511742 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.512157 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.512162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.512471 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:52.512838 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.512846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.513799 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.513800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.515126 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.515129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.515774 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.515775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.515937 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.516594 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:52.516606 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:52.518603 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:52.520906 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:52.520913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:52.520916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:52.521234 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:52.521483 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:52.521484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:52.521484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:52.521623 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:52.521978 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:52.521984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:52.522176 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.522955 (+ 779us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.523136 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260928 09:00:52.524122 11997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37765
I20260928 09:00:52.524219 11997 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:52.524411 11997 heartbeater.cc:511] Master 127.0.98.62:37765 requested a full tablet report, sending...
I20260928 09:00:52.524871 11647 ts_manager.cc:194] Registered new tserver with Master: bccf1bfa73e44d73882075146ac6d631 (127.0.98.2:46877)
I20260928 09:00:52.525418 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44185
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:00:52.571918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37765
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:00:52.705415 12002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:00:52.705781 12002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:00:52.705871 12002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:00:52.707600 12002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:00:52.707679 12002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:00:52.707724 12002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:00:52.709517 12002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:00:52.709784 12002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:00:52.710064 12002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:00:52.715070 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:52.715070 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:00:52.715447 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:00:52.715876 12002 server_base.cc:1034] running on GCE node
I20260928 09:00:52.716115 12002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:00:52.716358 12002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:00:52.717574 12002 hybrid_clock.cc:648] HybridClock initialized: now 1790586052717565 us; error 29 us; skew 500 ppm
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:00:52.720199 12002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:00:52.720703 12002 webserver.cc:460] Webserver started at http://127.0.98.3:43207/ using document root <none> and password file <none>
I20260928 09:00:52.720952 12002 fs_manager.cc:359] Metadata directory not provided
I20260928 09:00:52.721021 12002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:00:52.721154 12002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:00:52.733178 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "e380d3c211644e7f946cedb56b722be0"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "35346e3a0fd486e1080090f9054df23a"
server_key_iv: "788d69295e00eca99eb868a56992635d"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.733649 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e380d3c211644e7f946cedb56b722be0"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "35346e3a0fd486e1080090f9054df23a"
server_key_iv: "788d69295e00eca99eb868a56992635d"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.735121 12002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:52.755163 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.755406 12002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:00:52.755492 12002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "e380d3c211644e7f946cedb56b722be0"
format_stamp: "Formatted at 2026-09-28 09:00:52 on dist-test-slave-286n"
server_key: "35346e3a0fd486e1080090f9054df23a"
server_key_iv: "788d69295e00eca99eb868a56992635d"
server_key_version: "kuduclusterkey@0"
I20260928 09:00:52.755578 12002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:00:52.777643 12002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:00:52.780187 12002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:00:52.780393 12002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:00:52.780706 12002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:00:52.781128 12002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:00:52.781213 12002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.781283 12002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:00:52.781348 12002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:00:52.791087 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39901 every 8 connection(s)
I20260928 09:00:52.791252 12002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39901
I20260928 09:00:52.791762 12002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:00:52.793274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 12002
I20260928 09:00:52.793396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:00:52.793592 392 external_mini_cluster.cc:1428] Setting key 1f1e441025feaccb222abad32f67d810
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:00:52.813223 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.793088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48677 (local address 127.0.98.62:37765)
0928 09:00:52.793135 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.793137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.800939 (+ 7802us) server_negotiation.cc:407] Connection header received
0928 09:00:52.801622 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.801624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.801636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.801708 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:52.802532 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.803348 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.805394 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.805696 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.805735 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.808532 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:52.808533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:52.808534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:52.808541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:52.811499 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.812035 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.812036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.812037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.812104 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.812406 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.812407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.812407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.812546 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:52.812591 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.813036 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.813125 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:00:52.814036 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.792802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37765 (local address 127.0.98.3:48677)
0928 09:00:52.800747 (+ 7945us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.800811 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.801498 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.801856 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.801861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.802141 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:00:52.802419 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.802425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.803464 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.803464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.805308 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.805311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.805818 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.805819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.805970 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.806607 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:00:52.806617 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:00:52.808402 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:00:52.811603 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:52.811611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:52.811613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:52.811929 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:52.812184 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:00:52.812184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:00:52.812185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:00:52.812322 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:00:52.812665 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:00:52.812671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:00:52.812854 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.813610 (+ 756us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.813785 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7882,"thread_start_us":79,"threads_started":1}
I20260928 09:00:52.814711 12131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37765
I20260928 09:00:52.814851 12131 heartbeater.cc:464] Registering TS with master...
I20260928 09:00:52.815111 12131 heartbeater.cc:511] Master 127.0.98.62:37765 requested a full tablet report, sending...
I20260928 09:00:52.815734 11647 ts_manager.cc:194] Registered new tserver with Master: e380d3c211644e7f946cedb56b722be0 (127.0.98.3:39901)
I20260928 09:00:52.816282 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48677
I20260928 09:00:52.816944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:00:52.839529 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.822983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.98.62:37765)
0928 09:00:52.823034 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.823036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.823461 (+ 425us) server_negotiation.cc:407] Connection header received
0928 09:00:52.823579 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.823580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.823592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.823628 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:00:52.825647 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.826495 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.827755 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.828084 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.828134 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.836323 (+ 8189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:00:52.836325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:00:52.836325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:00:52.836334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:00:52.838165 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.838613 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.838615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.838615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.838674 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:00:52.838883 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:00:52.838883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:00:52.838884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:00:52.839026 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:00:52.839103 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.839345 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.839432 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:00:52.842470 11647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:00:52.844887 11647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:00:52.871405 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.863349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46877 (local address 127.0.0.1:45994)
0928 09:00:52.865248 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.865257 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.865298 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.865608 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.865611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.865613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:52.866031 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.866035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.867329 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.867330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.868967 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.868970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.871090 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.871091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.871175 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.871190 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.871262 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.871314 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1887,"thread_start_us":75,"threads_started":1}
I20260928 09:00:52.871904 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.863636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.0.98.2:46877)
0928 09:00:52.864820 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.864826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.865358 (+ 532us) server_negotiation.cc:407] Connection header received
0928 09:00:52.865407 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.865410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.865427 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.865499 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:52.866120 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.867220 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.869093 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.869856 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.869937 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.870013 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.871774 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.871820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":84,"threads_started":1}
I20260928 09:00:52.875010 11931 tablet_service.cc:1467] Processing CreateTablet for tablet 820916337cbf4138afd33c196fff160b (DEFAULT_TABLE table=test-table [id=fa74fa6a2f404906a46903a75b86e2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:00:52.875419 11931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820916337cbf4138afd33c196fff160b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:52.881299 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.862901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38637 (local address 127.0.0.1:43078)
0928 09:00:52.864898 (+ 1997us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.864946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.865020 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.870330 (+ 5310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.870331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.870334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:52.870376 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.870378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.873024 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.873025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.874058 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.874061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.881037 (+ 6976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.881038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.881107 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.881112 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.881167 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.881212 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":98,"threads_started":1}
I20260928 09:00:52.883493 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.863651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.0.98.1:38637)
0928 09:00:52.870103 (+ 6452us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.870108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.870120 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:00:52.870155 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.870157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.870174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.870254 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:52.870456 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.872917 (+ 2461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.879999 (+ 7082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.880860 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.880941 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.880998 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.883023 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.883073 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6410,"thread_start_us":59,"threads_started":1}
I20260928 09:00:52.884321 11793 tablet_service.cc:1467] Processing CreateTablet for tablet 820916337cbf4138afd33c196fff160b (DEFAULT_TABLE table=test-table [id=fa74fa6a2f404906a46903a75b86e2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:00:52.884909 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820916337cbf4138afd33c196fff160b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:52.892072 12149 tablet_bootstrap.cc:492] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Bootstrap starting.
I20260928 09:00:52.893340 12149 tablet_bootstrap.cc:654] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:52.893821 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.863151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39901 (local address 127.0.0.1:48014)
0928 09:00:52.865130 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.865139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.865174 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.879852 (+ 14678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.879854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.879857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:52.879901 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.879902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.889599 (+ 9697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.889600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.890798 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.890802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.893534 (+ 2732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.893535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.893615 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.893621 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.893659 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.893713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":74,"threads_started":1}
I20260928 09:00:52.894474 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.863268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.98.3:39901)
0928 09:00:52.879580 (+ 16312us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.879585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.879597 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:00:52.879639 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.879641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.879657 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.879744 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:52.881707 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.882905 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.890949 (+ 8044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.891728 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.894198 (+ 2470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.894309 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.894340 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.894389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16267,"thread_start_us":98,"threads_started":1}
I20260928 09:00:52.895612 12065 tablet_service.cc:1467] Processing CreateTablet for tablet 820916337cbf4138afd33c196fff160b (DEFAULT_TABLE table=test-table [id=fa74fa6a2f404906a46903a75b86e2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:00:52.895970 12065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820916337cbf4138afd33c196fff160b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:00:52.900015 12149 log.cc:824] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:52.900569 12150 tablet_bootstrap.cc:492] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: Bootstrap starting.
I20260928 09:00:52.901432 12150 tablet_bootstrap.cc:654] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:52.901660 12151 tablet_bootstrap.cc:492] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Bootstrap starting.
I20260928 09:00:52.902061 12150 log.cc:824] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:52.903736 12151 tablet_bootstrap.cc:654] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Neither blocks nor log segments found. Creating new log.
I20260928 09:00:52.904099 12151 log.cc:824] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Log is configured to *not* fsync() on all Append() calls
I20260928 09:00:52.904983 12149 tablet_bootstrap.cc:492] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: No bootstrap required, opened a new log
I20260928 09:00:52.905112 12149 ts_tablet_manager.cc:1397] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 09:00:52.907933 12149 raft_consensus.cc:348] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.908296 12149 raft_consensus.cc:374] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:52.908384 12149 raft_consensus.cc:729] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccf1bfa73e44d73882075146ac6d631, State: Initialized, Role: FOLLOWER
I20260928 09:00:52.908687 12149 consensus_queue.cc:260] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.909087 12149 ts_tablet_manager.cc:1428] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 09:00:52.909255 11997 heartbeater.cc:503] Master 127.0.98.62:37765 was elected leader, sending a full tablet report...
I20260928 09:00:52.911010 12150 tablet_bootstrap.cc:492] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: No bootstrap required, opened a new log
I20260928 09:00:52.911286 12150 ts_tablet_manager.cc:1397] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260928 09:00:52.914481 12150 raft_consensus.cc:348] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.914722 12150 raft_consensus.cc:374] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:52.914817 12150 raft_consensus.cc:729] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da368a08ea744c379e2925072a866b5b, State: Initialized, Role: FOLLOWER
I20260928 09:00:52.915020 12150 consensus_queue.cc:260] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.915359 12150 ts_tablet_manager.cc:1428] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 09:00:52.915519 11859 heartbeater.cc:503] Master 127.0.98.62:37765 was elected leader, sending a full tablet report...
I20260928 09:00:52.916896 12151 tablet_bootstrap.cc:492] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: No bootstrap required, opened a new log
I20260928 09:00:52.917032 12151 ts_tablet_manager.cc:1397] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260928 09:00:52.918598 12155 raft_consensus.cc:482] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:00:52.918712 12155 raft_consensus.cc:504] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.922976 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.919186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38637 (local address 127.0.98.2:40189)
0928 09:00:52.919303 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.919314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.919357 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.919671 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.919673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.919675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:52.919745 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.919747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.920791 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.920793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.921911 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.921914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.922700 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.922701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.922790 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.922794 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.922852 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.922898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 09:00:52.923390 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.919459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40189 (local address 127.0.98.1:38637)
0928 09:00:52.919506 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.919508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.919517 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:00:52.919544 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.919545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.919558 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.919617 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:52.919847 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.920487 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.922026 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.922570 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.922651 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.922663 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.923283 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.923328 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:00:52.923681 12151 raft_consensus.cc:348] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.923970 12151 raft_consensus.cc:374] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:00:52.924068 12151 raft_consensus.cc:729] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e380d3c211644e7f946cedb56b722be0, State: Initialized, Role: FOLLOWER
I20260928 09:00:52.924275 12151 consensus_queue.cc:260] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.925078 12151 ts_tablet_manager.cc:1428] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260928 09:00:52.927686 12131 heartbeater.cc:503] Master 127.0.98.62:37765 was elected leader, sending a full tablet report...
I20260928 09:00:52.928920 11810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820916337cbf4138afd33c196fff160b" candidate_uuid: "bccf1bfa73e44d73882075146ac6d631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da368a08ea744c379e2925072a866b5b" is_pre_election: true
I20260928 09:00:52.929083 11810 raft_consensus.cc:2454] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccf1bfa73e44d73882075146ac6d631 in term 0.
I20260928 09:00:52.929344 11887 leader_election.cc:302] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccf1bfa73e44d73882075146ac6d631, da368a08ea744c379e2925072a866b5b; no voters:
I20260928 09:00:52.929785 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.920092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37245 (local address 127.0.98.3:39901)
0928 09:00:52.920604 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.920606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.920616 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:00:52.920660 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.920661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.920674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.920730 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:00:52.925380 (+ 4650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.926057 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.928563 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.929151 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.929206 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.929218 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.929666 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.929708 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486}
I20260928 09:00:52.930195 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.919649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39901 (local address 127.0.98.2:37245)
0928 09:00:52.920161 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:52.920171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:00:52.920226 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:00:52.925192 (+ 4966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:00:52.925193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:00:52.925195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:00:52.925270 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.925272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.926409 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.926410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.927552 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:00:52.927555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.929403 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.929404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:00:52.929546 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.929552 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 09:00:52.930045 (+ 493us) client_negotiation.cc:240] Negotiation successful
0928 09:00:52.930093 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":86,"threads_started":1}
I20260928 09:00:52.930480 12156 raft_consensus.cc:2781] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:00:52.930610 12156 raft_consensus.cc:482] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:00:52.930697 12156 raft_consensus.cc:3037] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:52.930687 12082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820916337cbf4138afd33c196fff160b" candidate_uuid: "bccf1bfa73e44d73882075146ac6d631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e380d3c211644e7f946cedb56b722be0" is_pre_election: true
I20260928 09:00:52.930838 12082 raft_consensus.cc:2454] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bccf1bfa73e44d73882075146ac6d631 in term 0.
I20260928 09:00:52.931829 12156 raft_consensus.cc:504] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.932675 12156 leader_election.cc:288] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [CANDIDATE]: Term 1 election: Requested vote from peers da368a08ea744c379e2925072a866b5b (127.0.98.1:38637), e380d3c211644e7f946cedb56b722be0 (127.0.98.3:39901)
I20260928 09:00:52.932905 12082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820916337cbf4138afd33c196fff160b" candidate_uuid: "bccf1bfa73e44d73882075146ac6d631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e380d3c211644e7f946cedb56b722be0"
I20260928 09:00:52.932999 12082 raft_consensus.cc:3037] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:52.934228 12082 raft_consensus.cc:2454] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccf1bfa73e44d73882075146ac6d631 in term 1.
I20260928 09:00:52.934427 11810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820916337cbf4138afd33c196fff160b" candidate_uuid: "bccf1bfa73e44d73882075146ac6d631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da368a08ea744c379e2925072a866b5b"
I20260928 09:00:52.934551 11810 raft_consensus.cc:3037] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:00:52.935510 11810 raft_consensus.cc:2454] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bccf1bfa73e44d73882075146ac6d631 in term 1.
I20260928 09:00:52.935761 11887 leader_election.cc:302] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bccf1bfa73e44d73882075146ac6d631, da368a08ea744c379e2925072a866b5b; no voters:
I20260928 09:00:52.939949 12156 raft_consensus.cc:2781] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:00:52.940238 12156 raft_consensus.cc:686] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 1 LEADER]: Becoming Leader. State: Replica: bccf1bfa73e44d73882075146ac6d631, State: Running, Role: LEADER
I20260928 09:00:52.940397 12156 consensus_queue.cc:237] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } }
I20260928 09:00:52.940062 12155 leader_election.cc:288] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da368a08ea744c379e2925072a866b5b (127.0.98.1:38637), e380d3c211644e7f946cedb56b722be0 (127.0.98.3:39901)
I20260928 09:00:52.941710 11646 catalog_manager.cc:5495] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 reported cstate change: term changed from 0 to 1, leader changed from <none> to bccf1bfa73e44d73882075146ac6d631 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bccf1bfa73e44d73882075146ac6d631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bccf1bfa73e44d73882075146ac6d631" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 } health_report { overall_health: UNKNOWN } } }
Sep 28 09:00:52 dist-test-slave-286n krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:00:52.964365 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:52.960936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.0.98.2:46877)
0928 09:00:52.960983 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:00:52.960985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:00:52.961029 (+ 44us) server_negotiation.cc:407] Connection header received
0928 09:00:52.961075 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:00:52.961076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:00:52.961101 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:00:52.961181 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:00:52.961358 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.962081 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.963193 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:00:52.963518 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:00:52.963592 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:00:52.963834 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:00:52.963973 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:00:52.964059 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 09:00:52.964252 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 09:00:52.964277 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:00:52.966671 12082 raft_consensus.cc:1261] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 1 FOLLOWER]: Refusing update from remote peer bccf1bfa73e44d73882075146ac6d631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:00:52.967033 11810 raft_consensus.cc:1261] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b [term 1 FOLLOWER]: Refusing update from remote peer bccf1bfa73e44d73882075146ac6d631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:00:52.967447 12160 consensus_queue.cc:1035] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da368a08ea744c379e2925072a866b5b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:00:52.967602 12156 consensus_queue.cc:1035] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e380d3c211644e7f946cedb56b722be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 09:00:52.970285 11860 tablet.cc:2369] T 820916337cbf4138afd33c196fff160b P da368a08ea744c379e2925072a866b5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:00:52.973423 12163 mvcc.cc:205] Tried to move back new op lower bound from 7334240472948097024 to 7334240472845946880. Current Snapshot: MvccSnapshot[applied={T|T < 7334240472948097024}]
I20260928 09:00:52.973932 12165 mvcc.cc:205] Tried to move back new op lower bound from 7334240472948097024 to 7334240472845946880. Current Snapshot: MvccSnapshot[applied={T|T < 7334240472948097024}]
I20260928 09:00:52.975555 12164 mvcc.cc:205] Tried to move back new op lower bound from 7334240472948097024 to 7334240472845946880. Current Snapshot: MvccSnapshot[applied={T|T < 7334240472948097024}]
I20260928 09:00:52.980716 11646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:00:52.980825 11646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:00:52.981710 11646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a95b6b7f07b64f40bb3916680fa15c6d: Sending DeleteTablet for 3 replicas of tablet 820916337cbf4138afd33c196fff160b
I20260928 09:00:52.983775 11931 tablet_service.cc:1514] Processing DeleteTablet for tablet 820916337cbf4138afd33c196fff160b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45994
I20260928 09:00:52.984125 11793 tablet_service.cc:1514] Processing DeleteTablet for tablet 820916337cbf4138afd33c196fff160b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43078
I20260928 09:00:52.984297 12172 tablet_replica.cc:330] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: stopping tablet replica
I20260928 09:00:52.984381 12065 tablet_service.cc:1514] Processing DeleteTablet for tablet 820916337cbf4138afd33c196fff160b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:00:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48014
I20260928 09:00:52.984422 12172 raft_consensus.cc:2229] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:00:52.984583 12172 raft_consensus.cc:2258] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:00:52.985118 12172 ts_tablet_manager.cc:1905] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:00:52.985147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 11728
I20260928 09:00:52.988181 12172 ts_tablet_manager.cc:1918] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:00:52.988255 12172 log.cc:1196] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/820916337cbf4138afd33c196fff160b
I20260928 09:00:52.988572 12172 ts_tablet_manager.cc:1939] T 820916337cbf4138afd33c196fff160b P bccf1bfa73e44d73882075146ac6d631: Deleting consensus metadata
I20260928 09:00:52.989183 11633 catalog_manager.cc:4841] TS bccf1bfa73e44d73882075146ac6d631 (127.0.98.2:46877): tablet 820916337cbf4138afd33c196fff160b (table test-table [id=fa74fa6a2f404906a46903a75b86e2ca]) successfully deleted
I20260928 09:00:53.000809 12174 tablet_replica.cc:330] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: stopping tablet replica
I20260928 09:00:53.000993 12174 raft_consensus.cc:2229] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:00:53.001147 12174 raft_consensus.cc:2258] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:00:53.003095 12174 ts_tablet_manager.cc:1905] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:00:53.007771 12174 ts_tablet_manager.cc:1918] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:00:53.007860 12174 log.cc:1196] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/820916337cbf4138afd33c196fff160b
W20260928 09:00:53.010723 11634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38637 (error 108)
I20260928 09:00:53.010402 12174 ts_tablet_manager.cc:1939] T 820916337cbf4138afd33c196fff160b P e380d3c211644e7f946cedb56b722be0: Deleting consensus metadata
I20260928 09:00:53.012161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 11867
I20260928 09:00:53.013402 11634 catalog_manager.cc:4841] TS e380d3c211644e7f946cedb56b722be0 (127.0.98.3:39901): tablet 820916337cbf4138afd33c196fff160b (table test-table [id=fa74fa6a2f404906a46903a75b86e2ca]) successfully deleted
I20260928 09:00:53.013700 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:00:53.013575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38637 (local address 127.0.0.1:43084)
0928 09:00:53.013625 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 09:00:53.013662 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:00:53.013845 11634 catalog_manager.cc:4584] TS da368a08ea744c379e2925072a866b5b (127.0.98.1:38637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 820916337cbf4138afd33c196fff160b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38637: connect: Connection refused (error 111)
I20260928 09:00:53.018497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 12002
I20260928 09:00:53.023934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 11615
I20260928 09:00:53.033622 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 09:00:53.378088 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 09:00:53.378185 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 09:00:53.723006 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 09:00:53.723 UTC [11201] LOG: received smart shutdown request
2026-09-28 09:00:53.727 UTC [11201] LOG: background worker "logical replication launcher" (PID 11210) exited with exit code 1
2026-09-28 09:00:53.727 UTC [11205] LOG: shutting down
2026-09-28 09:00:53.920 UTC [11201] LOG: database system is shut down
2026-09-28T09:00:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47781 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:00:54 dist-test-slave-286n krb5kdc[12184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:00:54 dist-test-slave-286n krb5kdc[12184](info): set up 2 sockets
Sep 28 09:00:54 dist-test-slave-286n krb5kdc[12184](info): commencing operation
krb5kdc: starting...
W20260928 09:00:55.072788 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:00:55 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:00:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 09:00:55.238692 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaJshIb/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/postgres -l logfile start
2026-09-28 09:00:57.820 UTC [12214] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 09:00:57.820 UTC [12214] LOG: listening on IPv4 address "127.0.98.20", port 54483
2026-09-28 09:00:57.824 UTC [12214] LOG: listening on Unix socket "/tmp/.s.PGSQL.54483"
2026-09-28 09:00:57.852 UTC [12217] LOG: database system was shut down at 2026-09-28 09:00:56 UTC
2026-09-28 09:00:57.855 UTC [12214] LOG: database system is ready to accept connections
I20260928 09:00:58.812523 392 mini_postgres.cc:96] Postgres bound to 54483
2026-09-28 09:00:58.818 UTC [12225] FATAL: database "slave" does not exist
127.0.98.20:54483 - accepting connections
I20260928 09:00:58.819342 392 mini_ranger.cc:161] Starting Ranger...
I20260928 09:00:58.833724 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 09:00:59.338475 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 09:00:59.338567 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-admin
2026-09-28 09:00:59,604 [I] DB FLAVOR :POSTGRES
2026-09-28 09:00:59,604 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:00:59,604 [I] Checking connection..
2026-09-28 09:00:59,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 09:00:59,836 [I] Checking connection passed.
2026-09-28 09:00:59,836 [I] --------- Verifying version history table ---------
2026-09-28 09:00:59,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:01:00,066 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:01:00,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:01:00,322 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 09:01:00,322 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 09:01:00,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 09:01:00.600 UTC [12289] WARNING: there is no transaction in progress
2026-09-28 09:01:00,607 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 09:01:00,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 09:01:00,865 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 09:01:00,865 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:01:00,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 09:01:01,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 09:01:01,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-286n.c.gcp-upstream.internal','N') ;"
2026-09-28 09:01:01,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:01:01,860 [I] Table x_portal_user does not exist in database ranger
2026-09-28 09:01:01,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:01:02,145 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 09:01:02,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:01:02,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:01:02,618 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 09:01:02,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 09:01:03.639 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.655 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.672 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.687 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.704 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.725 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.791 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.798 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.814 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.825 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.836 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.850 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.861 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.867 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.874 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:03.880 UTC [12397] WARNING: there is no transaction in progress
2026-09-28 09:01:04,572 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 09:01:04,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 09:01:04,817 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 09:01:04,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 09:01:05,079 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 09:01:05,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 09:01:05,320 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 09:01:05,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 09:01:05,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 09:01:05,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-286n.c.gcp-upstream.internal';"
2026-09-28 09:01:06,059 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 09:01:06,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 09:01:06,304 [I] DB_PATCHES have already been applied
I20260928 09:01:06.310384 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaJshIb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 09:01:06.310482 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 09:01:06.313726 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37783
Sep 28, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 09:01:07.427539 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.608s user 0.000s sys 0.008s
Sep 28, 2026 9:01:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:01:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:01:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:01:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:01:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 09:01:28.814075 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 09:01:28 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586088, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 09:01:28.903292 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 09:01:28.912744 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 09:01:29.871135 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 09:01:29.871228 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms
I20260928 09:01:29.871250 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54483
2026-09-28 09:01:30,032 [I] DB FLAVOR :POSTGRES
2026-09-28 09:01:30,033 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 09:01:30,033 [I] Checking connection
2026-09-28 09:01:30,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 09:01:30,281 [I] connection success
2026-09-28 09:01:30,281 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 09:01:30,281 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 09:01:30,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 09:01:30,528 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 09:01:30,528 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 09:01:30,528 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 09:01:30,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54483/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 09:01:30,796 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 09:01:31.300341 392 mini_ranger_kms.cc:310] Created kms service
I20260928 09:01:31.470377 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 09:01:31.514009 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 09:01:31 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586088, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:31.878532 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 09:01:31.878643 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaJshIb/build/release/bin/postgresql.jar:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaJshIb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 09:01:31.878662 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 09:01:31.882251 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12577
I20260928 09:01:31.882341 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52897
Sep 28, 2026 9:01:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:01:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:01:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:01:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:01:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:01:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 09:01:34.186754 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.283s user 0.005s sys 0.139s
I20260928 09:01:34.186861 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 09:01:34.186906 392 mini_ranger_kms.cc:401] 127.0.98.20:52897/kms/v1/keys
Sep 28 09:01:34 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586094, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 09:01:34 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586094, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:01:35 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586094, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 09:01:37 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:01:37.365091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37425
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37425
--ranger_config_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:37.449251 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:37.449601 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:37.449630 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:37.450909 12628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:37.450959 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:37.450976 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:37.450991 12628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:37.451005 12628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:37.452726 12628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52897/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37425
--ranger_config_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:37.452979 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:37.453248 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:37.458448 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:37.458452 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:37.458453 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:37.458808 12628 server_base.cc:1034] running on GCE node
I20260928 09:01:37.459232 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:37.459554 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:37.460722 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790586097460697 us; error 41 us; skew 500 ppm
Sep 28 09:01:37 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:37.471164 12628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:37.471674 12628 webserver.cc:460] Webserver started at http://127.0.98.62:44291/ using document root <none> and password file <none>
I20260928 09:01:37.471927 12628 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:37.471990 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:37.472160 12628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:01:37 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:37.518388 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "a3abf042108e4c4aa9b155137b3269b8"
format_stamp: "Formatted at 2026-09-28 09:01:37 on dist-test-slave-286n"
server_key: "5975250954c6f9f58afd449c4fa11f0c"
server_key_iv: "162c762585381bcb7da3d86004f878ec"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:37.518828 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3abf042108e4c4aa9b155137b3269b8"
format_stamp: "Formatted at 2026-09-28 09:01:37 on dist-test-slave-286n"
server_key: "5975250954c6f9f58afd449c4fa11f0c"
server_key_iv: "162c762585381bcb7da3d86004f878ec"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:37.520378 12628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:01:37.534963 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:37.535202 12628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:37.535342 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "a3abf042108e4c4aa9b155137b3269b8"
format_stamp: "Formatted at 2026-09-28 09:01:37 on dist-test-slave-286n"
server_key: "5975250954c6f9f58afd449c4fa11f0c"
server_key_iv: "162c762585381bcb7da3d86004f878ec"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:37.535466 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:37.565752 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:37.567957 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:37.568156 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:37.573361 12628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37425
I20260928 09:01:37.573385 12695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37425 every 8 connection(s)
I20260928 09:01:37.573823 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:37.574506 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:37.576872 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Bootstrap starting.
I20260928 09:01:37.577464 12696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:37.577769 12696 log.cc:824] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:37.578470 12696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: No bootstrap required, opened a new log
I20260928 09:01:37.580588 12696 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:37.580811 12696 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:37.580838 12696 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3abf042108e4c4aa9b155137b3269b8, State: Initialized, Role: FOLLOWER
I20260928 09:01:37.581000 12696 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:37.581094 12696 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:01:37.581128 12696 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:01:37.581207 12696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:37.581585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 12628
I20260928 09:01:37.581691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:01:37.581895 392 external_mini_cluster.cc:1428] Setting key 735f0f237eecd3dfa0d76eb6658b3526
I20260928 09:01:37.581897 12696 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:37.581975 12696 leader_election.cc:302] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3abf042108e4c4aa9b155137b3269b8; no voters:
I20260928 09:01:37.582126 12696 leader_election.cc:288] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:01:37.582616 12696 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:37.582252 12699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:37.583148 12699 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 LEADER]: Becoming Leader. State: Replica: a3abf042108e4c4aa9b155137b3269b8, State: Running, Role: LEADER
I20260928 09:01:37.583307 12699 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:37.584560 12699 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3abf042108e4c4aa9b155137b3269b8. Latest consensus state: current_term: 1 leader_uuid: "a3abf042108e4c4aa9b155137b3269b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } } }
I20260928 09:01:37.584741 12699 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:37.585269 12715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:37.586001 12700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3abf042108e4c4aa9b155137b3269b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } } }
I20260928 09:01:37.586079 12700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:37.586474 12696 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 09:01:37.586948 12715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 09:01:37 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:37.592711 12715 catalog_manager.cc:1347] Generated new cluster ID: 150246e5c33949ff810a50dedd164960
I20260928 09:01:37.592864 12715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:01:37.596414 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:37.582427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.0.98.62:37425)
0928 09:01:37.582617 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:37.582626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:37.582882 (+ 256us) server_negotiation.cc:407] Connection header received
0928 09:01:37.583116 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:37.583126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:37.583196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:37.583370 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:37.584263 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:37.585618 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:37.586783 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:37.587388 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:37.587498 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:37.590954 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:37.591014 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:37.591014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:37.591122 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:37.594143 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:37.594574 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:37.594577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:37.594578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:37.594695 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:37.594912 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:37.594913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:37.594913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:37.595462 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:37.595683 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:37.595770 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 09:01:37.596136 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 09:01:37.605504 12715 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:01:37.606129 12715 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:01:37.608508 12715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Generated new TSK 0
I20260928 09:01:37.608747 12715 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:01:38.287940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37425
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:38.376776 12741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:38.377048 12741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:38.377089 12741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:38.378821 12741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:38.378880 12741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:38.378921 12741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:01:38.380497 12741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:38.380765 12741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:38.381011 12741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:38.386518 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:38.386543 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:38.386791 12741 server_base.cc:1034] running on GCE node
W20260928 09:01:38.386512 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:38.387033 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:38.387323 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:38.388484 12741 hybrid_clock.cc:648] HybridClock initialized: now 1790586098388471 us; error 28 us; skew 500 ppm
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:38.391191 12741 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:01:38.391610 12741 webserver.cc:460] Webserver started at http://127.0.98.1:34231/ using document root <none> and password file <none>
I20260928 09:01:38.391856 12741 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:38.391924 12741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:38.392076 12741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:38.408859 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "49ebef34d8e349fcbbdeb17669b2d672"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "f181908ccb0b560b130232665da60b59"
server_key_iv: "0ae02a8f64e1521066a7e726c17a46ad"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:38.409456 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "49ebef34d8e349fcbbdeb17669b2d672"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "f181908ccb0b560b130232665da60b59"
server_key_iv: "0ae02a8f64e1521066a7e726c17a46ad"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:38.411962 12741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:38.429373 12756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:38.429633 12741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:38.429735 12741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "49ebef34d8e349fcbbdeb17669b2d672"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "f181908ccb0b560b130232665da60b59"
server_key_iv: "0ae02a8f64e1521066a7e726c17a46ad"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:38.429834 12741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:38.483007 12741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:38.485718 12741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:38.485888 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:38.486132 12741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:38.486591 12741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:38.486644 12741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:38.486675 12741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:38.486725 12741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:38.494396 12741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45251
I20260928 09:01:38.494923 12741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:01:38.496057 12871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45251 every 8 connection(s)
I20260928 09:01:38.496423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 12741
I20260928 09:01:38.496529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:01:38.496769 392 external_mini_cluster.cc:1428] Setting key dbabbaa6e1217c213928184c778c2173
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:38.511860 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:38.496925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37425 (local address 127.0.98.1:59083)
0928 09:01:38.497400 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:38.497443 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:38.498067 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:38.501073 (+ 3006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:38.501078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:38.501349 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:38.501655 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:38.501663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.502894 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.502896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:38.504209 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:38.504212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.504889 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.504890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:38.505046 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:38.505801 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:38.505812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:38.507637 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:38.510037 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:38.510047 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:38.510050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:38.510383 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:38.510632 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:38.510633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:38.510633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:38.510773 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:38.511074 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:38.511087 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:38.511312 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 09:01:38.511420 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 09:01:38.511603 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":86,"threads_started":1}
I20260928 09:01:38.512717 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:38.496968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59083 (local address 127.0.98.62:37425)
0928 09:01:38.500796 (+ 3828us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:38.500799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:38.500811 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:01:38.500841 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:38.500841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:38.500855 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:38.501771 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:38.501822 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.502761 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.504326 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.504683 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.505364 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:38.507762 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:38.507764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:38.507764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:38.507773 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:38.509923 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:38.510483 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:38.510484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:38.510484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:38.510549 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:38.510847 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:38.510848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:38.510848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:38.510997 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:38.512438 (+ 1441us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:38.512483 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 09:01:38.512587 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3789,"thread_start_us":86,"threads_started":1}
I20260928 09:01:38.513068 12872 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37425
I20260928 09:01:38.513175 12872 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:38.513382 12872 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:38.514037 12660 ts_manager.cc:194] Registered new tserver with Master: 49ebef34d8e349fcbbdeb17669b2d672 (127.0.98.1:45251)
I20260928 09:01:38.514967 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59083
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:01:38.544288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37425
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:38.673739 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:38.674124 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:38.674206 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:38.676038 12878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:38.676159 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:38.676245 12878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:38.679296 12878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:38.679768 12878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:38.680073 12878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:38.686745 12885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:38.686849 12886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:38.687103 12888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:38.688185 12878 server_base.cc:1034] running on GCE node
I20260928 09:01:38.688512 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:38.688840 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:38.690011 12878 hybrid_clock.cc:648] HybridClock initialized: now 1790586098689989 us; error 36 us; skew 500 ppm
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:38.693320 12878 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:01:38.693899 12878 webserver.cc:460] Webserver started at http://127.0.98.2:36347/ using document root <none> and password file <none>
I20260928 09:01:38.694149 12878 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:38.694198 12878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:38.694324 12878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:38.714228 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "0997db36438247c6ae4c23f7ddfc3fd0"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "7eafdd15eaf676636b7c26013e84e6aa"
server_key_iv: "2f851556bc65da75868d28d9388c8862"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:38.714884 12878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0997db36438247c6ae4c23f7ddfc3fd0"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "7eafdd15eaf676636b7c26013e84e6aa"
server_key_iv: "2f851556bc65da75868d28d9388c8862"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:38.720557 12878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 09:01:38.743294 12895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:38.743563 12878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 09:01:38.743673 12878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "0997db36438247c6ae4c23f7ddfc3fd0"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "7eafdd15eaf676636b7c26013e84e6aa"
server_key_iv: "2f851556bc65da75868d28d9388c8862"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:38.743770 12878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:38.772843 12878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:38.775419 12878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:38.775667 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:38.775959 12878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:38.776386 12878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:38.776473 12878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:38.776540 12878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:38.776594 12878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:38.783385 12878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46635
I20260928 09:01:38.783917 12878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:01:38.788137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 12878
I20260928 09:01:38.788254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:01:38.788473 392 external_mini_cluster.cc:1428] Setting key 5485f73fc0dc5c4941560c2b14aecc80
I20260928 09:01:38.786676 13009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46635 every 8 connection(s)
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:38.801697 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:38.787746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59653 (local address 127.0.98.62:37425)
0928 09:01:38.787801 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:38.787805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:38.791143 (+ 3338us) server_negotiation.cc:407] Connection header received
0928 09:01:38.791872 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:38.791873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:38.791886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:38.791917 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:38.792807 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.793562 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.795035 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.795338 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.795382 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:38.798220 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:38.798222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:38.798222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:38.798230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:38.799914 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:38.800434 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:38.800435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:38.800436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:38.800496 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:38.800841 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:38.800842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:38.800842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:38.801002 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:38.801057 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:38.801469 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 09:01:38.801580 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:01:38.802522 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:38.787357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37425 (local address 127.0.98.2:59653)
0928 09:01:38.789213 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:38.791036 (+ 1823us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:38.791743 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:38.792106 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:38.792111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:38.792381 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:38.792696 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:38.792703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.793695 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.793696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:38.794932 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:38.794935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:38.795492 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:38.795492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:38.795641 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:38.796282 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:38.796293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:38.798097 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:38.800015 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:38.800024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:38.800027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:38.800345 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:38.800559 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:38.800560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:38.800561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:38.800750 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:38.801093 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:38.801099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:38.801303 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 09:01:38.802099 (+ 796us) client_negotiation.cc:240] Negotiation successful
0928 09:01:38.802277 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1791,"thread_start_us":112,"threads_started":1}
I20260928 09:01:38.804955 13010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37425
I20260928 09:01:38.805063 13010 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:38.805271 13010 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:38.805723 12660 ts_manager.cc:194] Registered new tserver with Master: 0997db36438247c6ae4c23f7ddfc3fd0 (127.0.98.2:46635)
I20260928 09:01:38.806206 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59653
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:01:38.839453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37425
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:38.966843 13015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:38.967235 13015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:38.967391 13015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:38.969411 13015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:38.969529 13015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:38.969636 13015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:01:38.972287 13015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:38.972877 13015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:38.973238 13015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:38.980794 13020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:38.981320 13021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:38.981662 13015 server_base.cc:1034] running on GCE node
W20260928 09:01:38.981951 13023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:38.982237 13015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:38.982642 13015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:38.984735 13015 hybrid_clock.cc:648] HybridClock initialized: now 1790586098984652 us; error 92 us; skew 500 ppm
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:38.988361 13015 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:01:38.989015 13015 webserver.cc:460] Webserver started at http://127.0.98.3:38229/ using document root <none> and password file <none>
I20260928 09:01:38.989315 13015 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:38.989390 13015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:38.989532 13015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 09:01:38 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:39.001534 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "2b7231322788481282aa2d41b28a74cb"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "eff9bf16389fbe0f7a31a0f6694bdffc"
server_key_iv: "e4dc44b6b751b94485dee90cd26ce317"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.002022 13015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2b7231322788481282aa2d41b28a74cb"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "eff9bf16389fbe0f7a31a0f6694bdffc"
server_key_iv: "e4dc44b6b751b94485dee90cd26ce317"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.003863 13015 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 09:01:39.021145 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.022454 13015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:39.022588 13015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "2b7231322788481282aa2d41b28a74cb"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "eff9bf16389fbe0f7a31a0f6694bdffc"
server_key_iv: "e4dc44b6b751b94485dee90cd26ce317"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.022686 13015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:39.040341 13015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:39.042953 13015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:39.043099 13015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:39.043344 13015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:39.043749 13015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:39.043807 13015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.043838 13015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:39.043888 13015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.050488 13015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45581
I20260928 09:01:39.050932 13015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:01:39.052305 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45581 every 8 connection(s)
I20260928 09:01:39.058986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13015
I20260928 09:01:39.059098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:01:39.059307 392 external_mini_cluster.cc:1428] Setting key c5d3953c12b59425501b8adc4361f5d6
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:39.063916 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.053296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53097 (local address 127.0.98.62:37425)
0928 09:01:39.053345 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:39.053347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:39.053538 (+ 191us) server_negotiation.cc:407] Connection header received
0928 09:01:39.054265 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:39.054266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:39.054278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:39.054309 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:39.055117 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.055891 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.057410 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.057705 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.057796 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.060410 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:39.060412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:39.060412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:39.060419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:39.062262 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.062760 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.062762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.062762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.062818 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.063084 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.063085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.063086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.063280 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:39.063328 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:39.063725 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 09:01:39.063816 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:01:39.064424 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.053050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37425 (local address 127.0.98.3:53097)
0928 09:01:39.053411 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:39.053456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:39.054126 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:39.054492 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:39.054497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:39.054739 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:39.055008 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.055015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.056011 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.056012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.057304 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.057307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.057834 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.057835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.057977 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.058632 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:39.058642 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:39.060319 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:39.062359 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.062366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.062368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.062666 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.062885 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.062886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.062886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.063010 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.063380 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:39.063386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:39.063560 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 09:01:39.064014 (+ 454us) client_negotiation.cc:240] Negotiation successful
0928 09:01:39.064189 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":80,"threads_started":1}
I20260928 09:01:39.065073 13144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37425
I20260928 09:01:39.065181 13144 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.065366 13144 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:39.065752 12660 ts_manager.cc:194] Registered new tserver with Master: 2b7231322788481282aa2d41b28a74cb (127.0.98.3:45581)
I20260928 09:01:39.066258 12660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53097
I20260928 09:01:39.071218 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:01:39.071894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 12741
I20260928 09:01:39.077462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 12878
I20260928 09:01:39.083526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13015
I20260928 09:01:39.088080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 12628
I20260928 09:01:39.094808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37425
--webserver_interface=127.0.98.62
--webserver_port=44291
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37425
--ranger_config_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:39.179073 13147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:39.179281 13147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:39.179304 13147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:39.180367 13147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:39.180402 13147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:39.180418 13147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:39.180429 13147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:39.180442 13147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:39.182040 13147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37425
--ranger_config_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:39.182236 13147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:39.182401 13147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:39.187352 13152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.187392 13155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.187392 13153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:39.187553 13147 server_base.cc:1034] running on GCE node
I20260928 09:01:39.187826 13147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:39.188067 13147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:39.189229 13147 hybrid_clock.cc:648] HybridClock initialized: now 1790586099189206 us; error 36 us; skew 500 ppm
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:39.191957 13147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:39.192553 13147 webserver.cc:460] Webserver started at http://127.0.98.62:44291/ using document root <none> and password file <none>
I20260928 09:01:39.192850 13147 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:39.192947 13147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:39.242203 13147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:39.244774 13163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.244961 13147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:39.245072 13147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "a3abf042108e4c4aa9b155137b3269b8"
format_stamp: "Formatted at 2026-09-28 09:01:37 on dist-test-slave-286n"
server_key: "5975250954c6f9f58afd449c4fa11f0c"
server_key_iv: "162c762585381bcb7da3d86004f878ec"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.245484 13147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:39.262763 13147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:39.265025 13147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:39.265194 13147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:39.269286 13147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37425
I20260928 09:01:39.269333 13215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37425 every 8 connection(s)
I20260928 09:01:39.269699 13147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:39.270870 13216 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:01:39.271554 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Bootstrap starting.
I20260928 09:01:39.271608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13147
I20260928 09:01:39.271915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45251
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37425
--webserver_port=34231
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:39.282111 13216 log.cc:824] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:39.284452 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:01:39.284794 13216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Bootstrap complete.
I20260928 09:01:39.288141 13216 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:39.288470 13216 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3abf042108e4c4aa9b155137b3269b8, State: Initialized, Role: FOLLOWER
I20260928 09:01:39.288625 13216 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:39.288717 13216 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:01:39.288754 13216 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:01:39.288800 13216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:01:39.290104 13216 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:39.290220 13216 leader_election.cc:302] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3abf042108e4c4aa9b155137b3269b8; no voters:
I20260928 09:01:39.290400 13216 leader_election.cc:288] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:01:39.290494 13220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:01:39.290632 13216 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:39.290683 13220 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [term 2 LEADER]: Becoming Leader. State: Replica: a3abf042108e4c4aa9b155137b3269b8, State: Running, Role: LEADER
I20260928 09:01:39.290782 13220 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } }
I20260928 09:01:39.291164 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3abf042108e4c4aa9b155137b3269b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } } }
I20260928 09:01:39.291242 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:39.291476 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3abf042108e4c4aa9b155137b3269b8. Latest consensus state: current_term: 2 leader_uuid: "a3abf042108e4c4aa9b155137b3269b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3abf042108e4c4aa9b155137b3269b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37425 } } }
I20260928 09:01:39.291601 13228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:39.292145 13228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:01:39.292265 13228 catalog_manager.cc:1259] Loaded cluster ID: 150246e5c33949ff810a50dedd164960
I20260928 09:01:39.292291 13228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:01:39.292517 13216 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 09:01:39.291599 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:39.296069 13228 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:01:39.296615 13228 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Loaded TSK: 0
I20260928 09:01:39.296947 13228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:01:39.394685 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:39.394989 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:39.395031 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:39.396487 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:39.396574 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:39.396624 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:01:39.398898 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34231
--tserver_master_addrs=127.0.98.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:39.399395 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:39.399720 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:39.406138 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.406464 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.406246 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:39.406661 13218 server_base.cc:1034] running on GCE node
I20260928 09:01:39.406913 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:39.407195 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:39.408360 13218 hybrid_clock.cc:648] HybridClock initialized: now 1790586099408339 us; error 37 us; skew 500 ppm
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:39.410939 13218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:01:39.411442 13218 webserver.cc:460] Webserver started at http://127.0.98.1:34231/ using document root <none> and password file <none>
I20260928 09:01:39.411669 13218 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:39.411751 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:39.435746 13218 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 09:01:39.436702 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.436888 13218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:39.436980 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "49ebef34d8e349fcbbdeb17669b2d672"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "f181908ccb0b560b130232665da60b59"
server_key_iv: "0ae02a8f64e1521066a7e726c17a46ad"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.437366 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:39.457885 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:39.460289 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:39.460460 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:39.460762 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:39.461149 13218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:39.461208 13218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.461243 13218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:39.461299 13218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.468511 13218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45251
I20260928 09:01:39.468601 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45251 every 8 connection(s)
I20260928 09:01:39.469040 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:01:39.470077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13218
I20260928 09:01:39.470350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46635
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37425
--webserver_port=36347
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:39.487582 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.469810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37425 (local address 127.0.98.1:52823)
0928 09:01:39.470182 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:39.470228 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:39.470772 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:39.471263 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:39.471267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:39.471483 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:39.472027 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.472035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.473820 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.473822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.475307 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.475310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.476311 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.476313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.476476 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.477175 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:39.477189 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:39.479891 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:39.485149 (+ 5258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.485160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.485163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.485806 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.486147 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.486147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.486148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.486292 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.486834 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:39.486841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:39.487023 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 09:01:39.487143 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 09:01:39.487318 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":112,"threads_started":1}
I20260928 09:01:39.488417 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.469975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52823 (local address 127.0.98.62:37425)
0928 09:01:39.470166 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:39.470177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:39.470359 (+ 182us) server_negotiation.cc:407] Connection header received
0928 09:01:39.471022 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:39.471027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:39.471076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:39.471207 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:39.472187 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.473681 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.475424 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.476176 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.476308 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.482413 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:39.482419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:39.482420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:39.482439 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:39.485024 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.485921 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.485925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.485925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.486062 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.486387 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.486387 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.486388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.486722 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:39.486804 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:39.488060 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0928 09:01:39.488194 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 09:01:39.489585 13378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37425
I20260928 09:01:39.489686 13378 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.489910 13378 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
W20260928 09:01:39.490906 13378 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37425 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 09:01:39.490979 13378 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.491055 13378 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:39.491575 13378 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.491650 13378 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
W20260928 09:01:39.492003 13378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:01:39.582188 13382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:39.582490 13382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:39.582526 13382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:39.584196 13382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:39.584281 13382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:39.584331 13382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:01:39.586283 13382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36347
--tserver_master_addrs=127.0.98.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:39.586550 13382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:39.586798 13382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:39.592566 13387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.592793 13390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.592583 13388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:39.593325 13382 server_base.cc:1034] running on GCE node
I20260928 09:01:39.593542 13382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:39.593835 13382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:39.595005 13382 hybrid_clock.cc:648] HybridClock initialized: now 1790586099594986 us; error 39 us; skew 500 ppm
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:39.597734 13382 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:01:39.598250 13382 webserver.cc:460] Webserver started at http://127.0.98.2:36347/ using document root <none> and password file <none>
I20260928 09:01:39.598492 13382 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:39.598573 13382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:39.619135 13382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:39.620146 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.620404 13382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:39.620527 13382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "0997db36438247c6ae4c23f7ddfc3fd0"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "7eafdd15eaf676636b7c26013e84e6aa"
server_key_iv: "2f851556bc65da75868d28d9388c8862"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.621111 13382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:39.641052 13382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:39.643024 13382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:39.643193 13382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:39.643471 13382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:39.643834 13382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:39.643893 13382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.643931 13382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:39.643957 13382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.651223 13382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46635
I20260928 09:01:39.651355 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46635 every 8 connection(s)
I20260928 09:01:39.651669 13382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:01:39.656658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13382
I20260928 09:01:39.657011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45581
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37425
--webserver_port=38229
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:39.663766 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.652589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37425 (local address 127.0.98.2:42827)
0928 09:01:39.652798 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:39.652841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:39.653352 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:39.653600 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:39.653609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:39.653792 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:39.654009 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.654014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.654835 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.654837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.656187 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.656192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.656835 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.656837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.656996 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.657928 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:39.657940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:39.659614 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:39.661759 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.661767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.661770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.662145 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.662401 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.662402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.662402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.662557 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.662912 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:39.662919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:39.663134 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 09:01:39.663258 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 09:01:39.663451 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":134,"threads_started":1}
I20260928 09:01:39.664139 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.652605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42827 (local address 127.0.98.62:37425)
0928 09:01:39.652698 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:39.652701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:39.652945 (+ 244us) server_negotiation.cc:407] Connection header received
0928 09:01:39.653457 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:39.653458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:39.653473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:39.653530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:39.654108 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.654703 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.656294 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.656587 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.656696 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.659724 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:39.659725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:39.659726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:39.659735 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:39.661651 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.662246 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.662248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.662248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.662324 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.662638 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.662639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.662639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.662803 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:39.662868 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:39.663948 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0928 09:01:39.664044 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 09:01:39.664580 13512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37425
I20260928 09:01:39.664742 13512 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.664990 13512 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
W20260928 09:01:39.665764 13512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37425 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 09:01:39.665908 13512 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.666028 13512 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:39.666414 13512 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.666531 13512 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
W20260928 09:01:39.666903 13512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:01:39.762074 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:39.762313 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:39.762356 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:39.763545 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:39.763616 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:39.763662 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:01:39.765218 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38229
--tserver_master_addrs=127.0.98.62:37425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:39.765486 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:39.765744 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:39.770857 13520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.770888 13521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:39.771268 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:39.771574 13515 server_base.cc:1034] running on GCE node
I20260928 09:01:39.771776 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:39.771996 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:39.773135 13515 hybrid_clock.cc:648] HybridClock initialized: now 1790586099773107 us; error 45 us; skew 500 ppm
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:39.776728 13515 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:01:39.777240 13515 webserver.cc:460] Webserver started at http://127.0.98.3:38229/ using document root <none> and password file <none>
I20260928 09:01:39.777493 13515 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:39.777572 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:39.801007 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:39.801774 13532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.801937 13515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:39.802039 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "2b7231322788481282aa2d41b28a74cb"
format_stamp: "Formatted at 2026-09-28 09:01:38 on dist-test-slave-286n"
server_key: "eff9bf16389fbe0f7a31a0f6694bdffc"
server_key_iv: "e4dc44b6b751b94485dee90cd26ce317"
server_key_version: "kuduclusterkey@0"
I20260928 09:01:39.802438 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:39.837098 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:39.839536 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:39.839706 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:39.839985 13515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:39.840408 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:39.840468 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.840505 13515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:39.840523 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:39.847457 13515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45581
I20260928 09:01:39.848053 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:01:39.848290 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45581 every 8 connection(s)
I20260928 09:01:39.851981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13515
Sep 28 09:01:39 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:39.862464 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.852417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.0.98.62:37425)
0928 09:01:39.852767 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:39.852769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:39.852775 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:01:39.852792 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:39.852793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:39.852804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:39.852858 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:39.853655 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.854382 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.855349 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.855641 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.855710 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.857216 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:39.857218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:39.857218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:39.857250 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:39.859383 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.859873 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.859874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.859874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.859951 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.861561 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.861561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.861562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.861667 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:39.861840 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:39.862259 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 09:01:39.862354 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":69,"threads_started":1}
I20260928 09:01:39.862823 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.848942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44791 (local address 127.0.98.62:37425)
0928 09:01:39.850210 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:39.850213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:39.850229 (+ 16us) server_negotiation.cc:407] Connection header received
0928 09:01:39.851087 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:39.851088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:39.851101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:39.851132 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:39.852061 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.852868 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.854458 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.854762 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.854853 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.858778 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:39.858780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:39.858780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:39.858788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:39.860421 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.860996 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.860997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.860997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.861055 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:39.861351 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:39.861351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:39.861352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:39.861492 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:39.861536 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:39.862673 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0928 09:01:39.862745 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235}
I20260928 09:01:39.863384 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:39.848967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37425 (local address 127.0.98.3:44791)
0928 09:01:39.849150 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:39.849195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:39.850948 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:39.851331 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:39.851336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:39.851600 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:39.851939 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.851947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.852975 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.852976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.854336 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:39.854339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:39.854888 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:39.854889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:39.855038 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:39.855863 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:39.855874 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:39.858671 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:39.860505 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.860514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.860517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.860884 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.861129 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:39.861130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:39.861130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:39.861276 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:39.861894 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:39.861900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:39.862103 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 09:01:39.862963 (+ 860us) client_negotiation.cc:240] Negotiation successful
0928 09:01:39.863141 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260928 09:01:39.864176 13646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37425
I20260928 09:01:39.864284 13646 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.864498 13646 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
W20260928 09:01:39.865201 13646 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37425 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 09:01:39.865293 13646 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.865367 13646 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:39.865770 13646 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:39.865862 13646 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
W20260928 09:01:39.866308 13646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 09:01:40 dist-test-slave-286n krb5kdc[12184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790586100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:40.492201 13378 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:40.492362 13378 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:40.492818 13180 ts_manager.cc:194] Registered new tserver with Master: 49ebef34d8e349fcbbdeb17669b2d672 (127.0.98.1:45251)
I20260928 09:01:40.493839 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52823
I20260928 09:01:40.667150 13512 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:40.667395 13512 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:40.667788 13180 ts_manager.cc:194] Registered new tserver with Master: 0997db36438247c6ae4c23f7ddfc3fd0 (127.0.98.2:46635)
I20260928 09:01:40.668344 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42827
Sep 28 09:01:40 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:40.866500 13646 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:40.866663 13646 heartbeater.cc:511] Master 127.0.98.62:37425 requested a full tablet report, sending...
I20260928 09:01:40.866971 13180 ts_manager.cc:194] Registered new tserver with Master: 2b7231322788481282aa2d41b28a74cb (127.0.98.3:45581)
I20260928 09:01:40.867489 13180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44791
I20260928 09:01:40.876829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:01:40.886276 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.877973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.98.62:37425)
0928 09:01:40.878265 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:40.878267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:40.878276 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:01:40.878303 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:40.878303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:40.878318 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:40.878413 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:40.879166 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.880014 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.881120 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.881473 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.881556 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.883059 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:40.883060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:40.883060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:40.883069 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:40.884938 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:40.885293 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:40.885294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:40.885295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:40.885363 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:40.885583 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:40.885584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:40.885584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:40.885710 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:40.885818 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:40.885955 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 09:01:40.886050 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":77,"threads_started":1}
I20260928 09:01:40.888486 13180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:01:40.890290 13180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:01:40.900245 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.893254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45251 (local address 127.0.0.1:55216)
0928 09:01:40.896765 (+ 3511us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:40.896776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:40.896854 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:40.897052 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:40.897055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:40.897058 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:40.897143 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:40.897145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.898009 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.898011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:40.899169 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:40.899172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.899930 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.899931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:40.900015 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.900029 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 09:01:40.900104 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 09:01:40.900157 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3484,"thread_start_us":89,"threads_started":1}
I20260928 09:01:40.900245 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.893490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.98.1:45251)
0928 09:01:40.893950 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:40.893955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:40.896868 (+ 2913us) server_negotiation.cc:407] Connection header received
0928 09:01:40.896928 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:40.896931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:40.896947 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:40.897041 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:40.897217 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.897920 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.899267 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.899816 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.899925 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.900013 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:40.900118 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 09:01:40.900161 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":53,"threads_started":1}
I20260928 09:01:40.901363 13312 tablet_service.cc:1467] Processing CreateTablet for tablet c46f51513f334ec29945fa2912869e6e (DEFAULT_TABLE table=test-table [id=344bae11073d4ed78bd360aa59395968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:40.901738 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46f51513f334ec29945fa2912869e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:40.904485 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.893667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.0.98.2:46635)
0928 09:01:40.894088 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:40.894092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:40.900512 (+ 6420us) server_negotiation.cc:407] Connection header received
0928 09:01:40.900619 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:40.900622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:40.900660 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:40.900719 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:40.900947 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.901953 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.903259 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.903988 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.904037 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.904117 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:40.904348 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 09:01:40.904403 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":62,"threads_started":1}
I20260928 09:01:40.904745 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.893471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46635 (local address 127.0.0.1:47560)
0928 09:01:40.900456 (+ 6985us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:40.900466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:40.900541 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:40.900836 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:40.900837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:40.900838 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:40.900867 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:40.900868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.902053 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.902053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:40.903153 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:40.903156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.904182 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.904183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:40.904250 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.904254 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:01:40.904596 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 09:01:40.904656 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6972,"thread_start_us":67,"threads_started":1}
I20260928 09:01:40.905467 13446 tablet_service.cc:1467] Processing CreateTablet for tablet c46f51513f334ec29945fa2912869e6e (DEFAULT_TABLE table=test-table [id=344bae11073d4ed78bd360aa59395968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:40.905827 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46f51513f334ec29945fa2912869e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:40.906692 13679 tablet_bootstrap.cc:492] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: Bootstrap starting.
I20260928 09:01:40.907370 13679 tablet_bootstrap.cc:654] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:40.907660 13679 log.cc:824] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:40.908468 13679 tablet_bootstrap.cc:492] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: No bootstrap required, opened a new log
I20260928 09:01:40.908558 13679 ts_tablet_manager.cc:1397] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:40.909020 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.893237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45581 (local address 127.0.0.1:41232)
0928 09:01:40.893616 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:40.893658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:40.893725 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:40.905182 (+ 11457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:40.905183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:40.905185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:40.905226 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:40.905228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.906589 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.906590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:40.907669 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:40.907673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.908740 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.908741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:40.908806 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.908809 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:01:40.908889 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:01:40.908933 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":86,"threads_started":1}
I20260928 09:01:40.909060 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:40.893325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.0.98.3:45581)
0928 09:01:40.905001 (+ 11676us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:40.905006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:40.905021 (+ 15us) server_negotiation.cc:407] Connection header received
0928 09:01:40.905059 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:40.905061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:40.905078 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:40.905168 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:40.905504 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.906479 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.907813 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:40.908575 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:40.908735 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:40.908813 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:40.908932 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 09:01:40.908978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11632,"thread_start_us":53,"threads_started":1}
I20260928 09:01:40.909696 13573 tablet_service.cc:1467] Processing CreateTablet for tablet c46f51513f334ec29945fa2912869e6e (DEFAULT_TABLE table=test-table [id=344bae11073d4ed78bd360aa59395968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:40.910009 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46f51513f334ec29945fa2912869e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:40.911134 13679 raft_consensus.cc:348] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:40.911293 13679 raft_consensus.cc:374] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:40.911335 13679 raft_consensus.cc:729] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ebef34d8e349fcbbdeb17669b2d672, State: Initialized, Role: FOLLOWER
I20260928 09:01:40.911466 13679 consensus_queue.cc:260] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:40.911761 13378 heartbeater.cc:503] Master 127.0.98.62:37425 was elected leader, sending a full tablet report...
I20260928 09:01:40.911755 13679 ts_tablet_manager.cc:1428] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:01:40.912207 13681 tablet_bootstrap.cc:492] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: Bootstrap starting.
I20260928 09:01:40.913290 13681 tablet_bootstrap.cc:654] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:40.913440 13683 tablet_bootstrap.cc:492] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Bootstrap starting.
I20260928 09:01:40.913631 13681 log.cc:824] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:40.914177 13683 tablet_bootstrap.cc:654] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:40.914438 13681 tablet_bootstrap.cc:492] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: No bootstrap required, opened a new log
I20260928 09:01:40.914508 13681 ts_tablet_manager.cc:1397] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:40.914636 13683 log.cc:824] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:40.915469 13683 tablet_bootstrap.cc:492] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: No bootstrap required, opened a new log
I20260928 09:01:40.915529 13683 ts_tablet_manager.cc:1397] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:01:40.917336 13681 raft_consensus.cc:348] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:40.917536 13681 raft_consensus.cc:374] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:40.917572 13681 raft_consensus.cc:729] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0997db36438247c6ae4c23f7ddfc3fd0, State: Initialized, Role: FOLLOWER
I20260928 09:01:40.917707 13681 consensus_queue.cc:260] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:40.917973 13681 ts_tablet_manager.cc:1428] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:01:40.918081 13512 heartbeater.cc:503] Master 127.0.98.62:37425 was elected leader, sending a full tablet report...
I20260928 09:01:40.918150 13683 raft_consensus.cc:348] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:40.918282 13683 raft_consensus.cc:374] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:40.918313 13683 raft_consensus.cc:729] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7231322788481282aa2d41b28a74cb, State: Initialized, Role: FOLLOWER
I20260928 09:01:40.918682 13683 consensus_queue.cc:260] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:40.919030 13683 ts_tablet_manager.cc:1428] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:01:40.919221 13646 heartbeater.cc:503] Master 127.0.98.62:37425 was elected leader, sending a full tablet report...
W20260928 09:01:40.970477 13379 tablet.cc:2369] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 09:01:41 dist-test-slave-286n krb5kdc[12184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 09:01:41.075065 13687 raft_consensus.cc:482] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:41.075232 13687 raft_consensus.cc:504] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:41.075641 13687 leader_election.cc:288] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ebef34d8e349fcbbdeb17669b2d672 (127.0.98.1:45251), 0997db36438247c6ae4c23f7ddfc3fd0 (127.0.98.2:46635)
I20260928 09:01:41.080104 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.076172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53171 (local address 127.0.98.2:46635)
0928 09:01:41.076217 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:41.076219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:41.076359 (+ 140us) server_negotiation.cc:407] Connection header received
0928 09:01:41.076473 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:41.076474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:41.076487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:41.076519 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:41.076802 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.077626 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.079076 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.079676 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.079754 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:41.079764 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:41.079954 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 09:01:41.080015 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:01:41.080377 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.075873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41035 (local address 127.0.98.1:45251)
0928 09:01:41.075931 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:41.075934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:41.076358 (+ 424us) server_negotiation.cc:407] Connection header received
0928 09:01:41.076473 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:41.076474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:41.076488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:41.076519 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:41.077073 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.077581 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.079092 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.079554 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.079589 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:41.079598 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:41.080267 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 09:01:41.080313 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:01:41.080545 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.076084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46635 (local address 127.0.98.3:53171)
0928 09:01:41.076290 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:41.076302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:41.076390 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:41.076590 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:41.076590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:41.076592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:41.076715 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:41.076717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.077720 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.077721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:41.078971 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:41.078974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.079774 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.079775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:41.079857 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:41.079862 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:01:41.080380 (+ 518us) client_negotiation.cc:240] Negotiation successful
0928 09:01:41.080432 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260928 09:01:41.080888 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.075846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45251 (local address 127.0.98.3:41035)
0928 09:01:41.076300 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:41.076310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:41.076390 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:41.076919 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:41.076920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:41.076923 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:41.076985 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:41.076986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.077960 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.077960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:41.078997 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:41.078999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.079892 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.079893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:41.079965 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:41.079968 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:01:41.080776 (+ 808us) client_negotiation.cc:240] Negotiation successful
0928 09:01:41.080821 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260928 09:01:41.080893 13466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46f51513f334ec29945fa2912869e6e" candidate_uuid: "2b7231322788481282aa2d41b28a74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" is_pre_election: true
I20260928 09:01:41.081056 13466 raft_consensus.cc:2454] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7231322788481282aa2d41b28a74cb in term 0.
I20260928 09:01:41.081302 13534 leader_election.cc:302] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0997db36438247c6ae4c23f7ddfc3fd0, 2b7231322788481282aa2d41b28a74cb; no voters:
I20260928 09:01:41.081620 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46f51513f334ec29945fa2912869e6e" candidate_uuid: "2b7231322788481282aa2d41b28a74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ebef34d8e349fcbbdeb17669b2d672" is_pre_election: true
I20260928 09:01:41.081768 13332 raft_consensus.cc:2454] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7231322788481282aa2d41b28a74cb in term 0.
I20260928 09:01:41.083014 13687 raft_consensus.cc:2781] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:01:41.083093 13687 raft_consensus.cc:482] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:01:41.083129 13687 raft_consensus.cc:3037] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:41.083827 13687 raft_consensus.cc:504] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:41.083969 13687 leader_election.cc:288] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [CANDIDATE]: Term 1 election: Requested vote from peers 49ebef34d8e349fcbbdeb17669b2d672 (127.0.98.1:45251), 0997db36438247c6ae4c23f7ddfc3fd0 (127.0.98.2:46635)
I20260928 09:01:41.084172 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46f51513f334ec29945fa2912869e6e" candidate_uuid: "2b7231322788481282aa2d41b28a74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ebef34d8e349fcbbdeb17669b2d672"
I20260928 09:01:41.084224 13466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46f51513f334ec29945fa2912869e6e" candidate_uuid: "2b7231322788481282aa2d41b28a74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0997db36438247c6ae4c23f7ddfc3fd0"
I20260928 09:01:41.084255 13332 raft_consensus.cc:3037] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:41.084292 13466 raft_consensus.cc:3037] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:41.085132 13332 raft_consensus.cc:2454] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7231322788481282aa2d41b28a74cb in term 1.
I20260928 09:01:41.085132 13466 raft_consensus.cc:2454] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7231322788481282aa2d41b28a74cb in term 1.
I20260928 09:01:41.085301 13534 leader_election.cc:302] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0997db36438247c6ae4c23f7ddfc3fd0, 2b7231322788481282aa2d41b28a74cb; no voters:
I20260928 09:01:41.085419 13687 raft_consensus.cc:2781] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:41.085603 13687 raft_consensus.cc:686] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 1 LEADER]: Becoming Leader. State: Replica: 2b7231322788481282aa2d41b28a74cb, State: Running, Role: LEADER
I20260928 09:01:41.085705 13687 consensus_queue.cc:237] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } }
I20260928 09:01:41.086548 13180 catalog_manager.cc:5495] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b7231322788481282aa2d41b28a74cb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2b7231322788481282aa2d41b28a74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7231322788481282aa2d41b28a74cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 } health_report { overall_health: UNKNOWN } } }
W20260928 09:01:41.099360 13647 tablet.cc:2369] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:01:41.128954 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.125165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.0.98.3:45581)
0928 09:01:41.125221 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:41.125224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:41.125280 (+ 56us) server_negotiation.cc:407] Connection header received
0928 09:01:41.125372 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:41.125373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:41.125422 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:41.125510 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:01:41.125675 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.126409 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.127661 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:41.127999 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:41.128073 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:41.128300 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:01:41.128448 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:01:41.128537 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:41.128825 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 09:01:41.128863 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 09:01:41.140829 13466 raft_consensus.cc:1261] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 1 FOLLOWER]: Refusing update from remote peer 2b7231322788481282aa2d41b28a74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:41.140982 13332 raft_consensus.cc:1261] T c46f51513f334ec29945fa2912869e6e P 49ebef34d8e349fcbbdeb17669b2d672 [term 1 FOLLOWER]: Refusing update from remote peer 2b7231322788481282aa2d41b28a74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:41.141247 13687 consensus_queue.cc:1035] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0997db36438247c6ae4c23f7ddfc3fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:41.141559 13687 consensus_queue.cc:1035] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ebef34d8e349fcbbdeb17669b2d672" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:41.147358 13695 mvcc.cc:205] Tried to move back new op lower bound from 7334240670269202432 to 7334240670047703040. Current Snapshot: MvccSnapshot[applied={T|T < 7334240670269202432}]
I20260928 09:01:41.147480 13696 mvcc.cc:205] Tried to move back new op lower bound from 7334240670269202432 to 7334240670047703040. Current Snapshot: MvccSnapshot[applied={T|T < 7334240670269202432}]
I20260928 09:01:41.149775 13698 mvcc.cc:205] Tried to move back new op lower bound from 7334240670269202432 to 7334240670047703040. Current Snapshot: MvccSnapshot[applied={T|T < 7334240670269202432}]
I20260928 09:01:41.151835 13179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:01:41.151943 13179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:01:41.152751 13179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3abf042108e4c4aa9b155137b3269b8: Sending DeleteTablet for 3 replicas of tablet c46f51513f334ec29945fa2912869e6e
I20260928 09:01:41.153103 13573 tablet_service.cc:1514] Processing DeleteTablet for tablet c46f51513f334ec29945fa2912869e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41232
I20260928 09:01:41.153569 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet c46f51513f334ec29945fa2912869e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55216
I20260928 09:01:41.153838 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet c46f51513f334ec29945fa2912869e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47560
I20260928 09:01:41.154322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13218
I20260928 09:01:41.164775 13704 tablet_replica.cc:330] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: stopping tablet replica
I20260928 09:01:41.164948 13704 raft_consensus.cc:2229] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:01:41.165102 13704 raft_consensus.cc:2258] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 09:01:41.169507 13165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45251 (error 108)
I20260928 09:01:41.169988 13704 ts_tablet_manager.cc:1905] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:41.171800 13704 ts_tablet_manager.cc:1918] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:01:41.171900 13704 log.cc:1196] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/c46f51513f334ec29945fa2912869e6e
I20260928 09:01:41.172196 13704 ts_tablet_manager.cc:1939] T c46f51513f334ec29945fa2912869e6e P 2b7231322788481282aa2d41b28a74cb: Deleting consensus metadata
I20260928 09:01:41.172434 13706 tablet_replica.cc:330] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: stopping tablet replica
I20260928 09:01:41.172578 13706 raft_consensus.cc:2229] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:01:41.172896 13706 raft_consensus.cc:2258] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:41.173368 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.173218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45251 (local address 127.0.0.1:55222)
0928 09:01:41.173272 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:41.173327 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 09:01:41.173511 13165 catalog_manager.cc:4584] TS 49ebef34d8e349fcbbdeb17669b2d672 (127.0.98.1:45251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c46f51513f334ec29945fa2912869e6e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45251: connect: Connection refused (error 111)
I20260928 09:01:41.173810 13166 catalog_manager.cc:4841] TS 2b7231322788481282aa2d41b28a74cb (127.0.98.3:45581): tablet c46f51513f334ec29945fa2912869e6e (table test-table [id=344bae11073d4ed78bd360aa59395968]) successfully deleted
I20260928 09:01:41.174178 13706 ts_tablet_manager.cc:1905] T c46f51513f334ec29945fa2912869e6e P 0997db36438247c6ae4c23f7ddfc3fd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:41.175809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13382
I20260928 09:01:41.180390 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:41.180256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46635 (local address 127.0.0.1:47566)
0928 09:01:41.180318 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:41.180346 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 09:01:41.181277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13515
I20260928 09:01:41.192776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13147
I20260928 09:01:41.198675 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 09:01:41.543491 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 09:01:41.543618 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 09:01:41.888451 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 09:01:41.888 UTC [12214] LOG: received smart shutdown request
2026-09-28 09:01:41.892 UTC [12214] LOG: background worker "logical replication launcher" (PID 12223) exited with exit code 1
2026-09-28 09:01:41.892 UTC [12218] LOG: shutting down
2026-09-28 09:01:42.071 UTC [12214] LOG: database system is shut down
2026-09-28T09:01:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48159 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:01:42 dist-test-slave-286n krb5kdc[13716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:01:42 dist-test-slave-286n krb5kdc[13716](info): set up 2 sockets
Sep 28 09:01:42 dist-test-slave-286n krb5kdc[13716](info): commencing operation
krb5kdc: starting...
W20260928 09:01:43.240626 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:01:43.361876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39955,127.0.98.61:43903,127.0.98.60:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:43.444236 13729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:43.444443 13729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:43.444466 13729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:43.445683 13729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:43.445744 13729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:43.445760 13729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:43.445772 13729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:43.445827 13729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:43.447367 13729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39955,127.0.98.61:43903,127.0.98.60:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:43.447625 13729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:43.447873 13729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:43.452759 13735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:43.452787 13734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:43.452800 13737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:43.452962 13729 server_base.cc:1034] running on GCE node
I20260928 09:01:43.453234 13729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:43.453536 13729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:43.454716 13729 hybrid_clock.cc:648] HybridClock initialized: now 1790586103454686 us; error 49 us; skew 500 ppm
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:43.462656 13729 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:43.463239 13729 webserver.cc:460] Webserver started at http://127.0.98.62:36051/ using document root <none> and password file <none>
I20260928 09:01:43.463512 13729 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:43.463587 13729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:43.463742 13729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:43.464928 13729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "c54db4ddbd7741f28f4838af1a260b5c"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "c8b18c598691856fa226acb9a5c708a1"
server_key_iv: "11f77db0c672d035cdd5cac1426643c1"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.465299 13729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "c54db4ddbd7741f28f4838af1a260b5c"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "c8b18c598691856fa226acb9a5c708a1"
server_key_iv: "11f77db0c672d035cdd5cac1426643c1"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.466671 13729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:01:43.467404 13744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:43.467585 13729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:01:43.467690 13729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "c54db4ddbd7741f28f4838af1a260b5c"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "c8b18c598691856fa226acb9a5c708a1"
server_key_iv: "11f77db0c672d035cdd5cac1426643c1"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.467773 13729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:43.510962 13729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:43.513677 13729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:43.513855 13729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:43.517925 13729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39955
I20260928 09:01:43.517975 13796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39955 every 8 connection(s)
I20260928 09:01:43.518342 13729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:43.519184 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:43.520769 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:43.527391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13729
I20260928 09:01:43.527511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:01:43.527799 392 external_mini_cluster.cc:1428] Setting key e29ba673acbbaf45880c86938fed228b
I20260928 09:01:43.530701 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.521417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.0.1:47322)
0928 09:01:43.521649 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.521677 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.522148 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.522410 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.522413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.522679 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.522868 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.522872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.523934 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.523934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.524601 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.524603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.525113 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.525114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.525188 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.525702 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.525711 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.527114 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.529124 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.529130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.529132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.529338 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.529603 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.529604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.529604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.529703 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.530092 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.530104 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.530169 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.530281 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.530522 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":97,"thread_start_us":87,"threads_started":1}
I20260928 09:01:43.530706 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.521449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.0.98.62:39955)
0928 09:01:43.521593 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.521601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.521802 (+ 201us) server_negotiation.cc:407] Connection header received
0928 09:01:43.522281 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.522283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.522300 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.522379 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.522987 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.523846 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.524722 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.525039 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.525119 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.527198 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.527206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.527206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.527213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.529022 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.529441 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.529444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.529444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.529529 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.529784 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.529785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.529785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.529995 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.530114 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.530320 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.530455 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 09:01:43.531559 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:01:43.531922 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.531801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59232)
0928 09:01:43.531850 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.531899 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:01:43.532097 13745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
I20260928 09:01:43.532519 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.532447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59238)
0928 09:01:43.532474 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.532501 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 09:01:43.532672 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43903: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 09:01:43.556110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43903
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39955,127.0.98.61:43903,127.0.98.60:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:43.581894 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } attempt: 1
I20260928 09:01:43.582429 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.582261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59244)
0928 09:01:43.582332 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.582389 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 09:01:43.582803 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.582703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59252)
0928 09:01:43.582753 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.582783 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 09:01:43.582928 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43903: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
W20260928 09:01:43.637545 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:43.637763 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:43.637785 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:43.638938 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:43.638983 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:43.638998 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:43.639010 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:43.639022 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:43.640545 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39955,127.0.98.61:43903,127.0.98.60:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:43.640811 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:43.640990 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:43.646014 13810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:43.646070 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:43.646052 13804 server_base.cc:1034] running on GCE node
W20260928 09:01:43.646029 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:43.646482 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:43.646765 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:43.647933 13804 hybrid_clock.cc:648] HybridClock initialized: now 1790586103647914 us; error 42 us; skew 500 ppm
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:43.650730 13804 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 09:01:43.651206 13804 webserver.cc:460] Webserver started at http://127.0.98.61:38647/ using document root <none> and password file <none>
I20260928 09:01:43.651118 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } attempt: 2
I20260928 09:01:43.651456 13804 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:43.651525 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:43.651583 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.651423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59260)
0928 09:01:43.651492 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.651545 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 09:01:43.651655 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:43.651937 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.651826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59270)
0928 09:01:43.651872 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.651902 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 09:01:43.652035 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43903: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43903: connect: Connection refused (error 111)
I20260928 09:01:43.652917 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data/instance:
uuid: "4f8b434a80184f7a83d5e457fc3cc89d"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "6d68dcb5288933808aa437ce7ee0ee6e"
server_key_iv: "74a0cf814172f4ab92e07299e9a9da94"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.653275 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal/instance:
uuid: "4f8b434a80184f7a83d5e457fc3cc89d"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "6d68dcb5288933808aa437ce7ee0ee6e"
server_key_iv: "74a0cf814172f4ab92e07299e9a9da94"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.654614 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:01:43.655366 13819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:43.655592 13804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:43.655676 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal
uuid: "4f8b434a80184f7a83d5e457fc3cc89d"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "6d68dcb5288933808aa437ce7ee0ee6e"
server_key_iv: "74a0cf814172f4ab92e07299e9a9da94"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.655776 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:43.677595 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:43.680003 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:43.680119 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:43.684366 13804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43903
I20260928 09:01:43.684432 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43903 every 8 connection(s)
I20260928 09:01:43.684810 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:01:43.685652 13872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:43.687247 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:01:43.691354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13804
I20260928 09:01:43.691455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-1/wal/instance
I20260928 09:01:43.691653 392 external_mini_cluster.cc:1428] Setting key 4742f69f02a319aaa08e1de454cac444
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:43.696591 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.688003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.0.98.62:39955)
0928 09:01:43.688083 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.688085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.688228 (+ 143us) server_negotiation.cc:407] Connection header received
0928 09:01:43.688720 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.688721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.688735 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.688789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.689366 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.689889 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.690874 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.691076 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.691131 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.693429 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.693430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.693431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.693439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.695001 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.695568 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.695569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.695569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.695621 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.695927 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.695928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.695928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.696035 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.696087 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.696455 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.696525 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 09:01:43.696988 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.687940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.0.1:47324)
0928 09:01:43.688086 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.688123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.688585 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.688846 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.688849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.689090 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.689294 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.689299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.689971 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.689972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.690792 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.690795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.691158 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.691159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.691258 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.691843 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.691851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.693313 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.695126 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.695134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.695137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.695485 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.695700 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.695701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.695702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.695857 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.696107 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.696115 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.696301 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.696434 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.696627 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260928 09:01:43.697880 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:01:43.706595 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.698276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.98.61:43903)
0928 09:01:43.698455 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.698460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.698485 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:01:43.698509 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.698511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.698526 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.698616 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.698870 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.700054 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.700767 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.701062 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.701176 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.703517 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.703524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.703525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.703532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.705275 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.705675 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.705678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.705678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.705730 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.705963 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.705964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.705964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.706099 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.706161 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.706368 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.706484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260928 09:01:43.706640 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.698202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59282)
0928 09:01:43.698280 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.698292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.698340 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.698623 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.698624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.698742 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.698792 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.698793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.700127 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.700128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.700615 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.700616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.701171 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.701172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.701236 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.701678 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.701683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.703433 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.705361 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.705362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.705362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.705592 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.705829 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.705830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.705830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.705900 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.706173 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.706174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.706224 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.706301 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.706418 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 09:01:43.707361 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:01:43.707765 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.707614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43436)
0928 09:01:43.707677 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.707734 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 09:01:43.707937 13822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
I20260928 09:01:43.708397 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.708277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43448)
0928 09:01:43.708329 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.708367 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:01:43.708521 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 09:01:43.719521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41479
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39955,127.0.98.61:43903,127.0.98.60:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:43.743199 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } attempt: 3
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:01:43.750984 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.743760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.98.61:43903)
0928 09:01:43.743837 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.743839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.743851 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:01:43.743942 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.743943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.743953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.744029 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.744356 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.744975 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.745695 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.745911 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.746000 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.748105 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.748107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.748107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.748117 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.749608 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.750051 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.750052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.750053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.750115 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.750365 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.750366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.750366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.750490 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.750572 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.750794 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.750880 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:01:43.750998 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.743664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59296)
0928 09:01:43.743734 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.743799 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.743859 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.744035 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.744035 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.744199 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.744267 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.744270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.745083 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.745084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.745607 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.745608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.745998 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.745999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.746101 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.746612 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.746616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.748016 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.749751 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.749752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.749752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.749942 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.750224 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.750225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.750226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.750294 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.750562 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.750563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.750613 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.750767 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.750878 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 09:01:43.751375 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:01:43.751765 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.751653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43462)
0928 09:01:43.751706 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.751735 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 09:01:43.752141 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.752054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43476)
0928 09:01:43.752098 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.752126 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 09:01:43.752249 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
I20260928 09:01:43.757754 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } attempt: 1
I20260928 09:01:43.758167 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.758047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43486)
0928 09:01:43.758102 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.758139 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:01:43.758514 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.758423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43498)
0928 09:01:43.758468 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.758495 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 09:01:43.758612 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
I20260928 09:01:43.783401 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } attempt: 1
I20260928 09:01:43.783932 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.783794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43506)
0928 09:01:43.783858 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.783902 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 09:01:43.784339 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.784254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43518)
0928 09:01:43.784296 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.784324 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:01:43.784459 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
W20260928 09:01:43.802933 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:43.803148 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:43.803169 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:43.804342 13879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:43.804382 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:43.804397 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:43.804409 13879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:43.804421 13879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:43.805887 13879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39955,127.0.98.61:43903,127.0.98.60:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:43.806082 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:43.806258 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:43.811023 13885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:43.811040 13884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:43.811177 13887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:43.811492 13879 server_base.cc:1034] running on GCE node
I20260928 09:01:43.811685 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:43.811920 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:43.813058 13879 hybrid_clock.cc:648] HybridClock initialized: now 1790586103813036 us; error 42 us; skew 500 ppm
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:43.815619 13879 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 09:01:43.816072 13879 webserver.cc:460] Webserver started at http://127.0.98.60:42673/ using document root <none> and password file <none>
I20260928 09:01:43.816305 13879 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:43.816375 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:43.816529 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:43.817621 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data/instance:
uuid: "4a7c6dde80ad4ce9972a913529631c65"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "b6a1ca78289877d301b24151ea6dbdbe"
server_key_iv: "41a50fbfb76e3e96f0dcd90ba41f0b4d"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.817983 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal/instance:
uuid: "4a7c6dde80ad4ce9972a913529631c65"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "b6a1ca78289877d301b24151ea6dbdbe"
server_key_iv: "41a50fbfb76e3e96f0dcd90ba41f0b4d"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.819347 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:01:43.820083 13894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:43.820272 13879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:43.820370 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal
uuid: "4a7c6dde80ad4ce9972a913529631c65"
format_stamp: "Formatted at 2026-09-28 09:01:43 on dist-test-slave-286n"
server_key: "b6a1ca78289877d301b24151ea6dbdbe"
server_key_iv: "41a50fbfb76e3e96f0dcd90ba41f0b4d"
server_key_version: "encryptionkey@0"
I20260928 09:01:43.820453 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:43.826748 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } attempt: 2
I20260928 09:01:43.827180 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.827049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43526)
0928 09:01:43.827107 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.827148 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:01:43.827561 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.827463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43538)
0928 09:01:43.827510 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.827538 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 09:01:43.827679 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41479: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41479: connect: Connection refused (error 111)
I20260928 09:01:43.836977 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:43.839411 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:43.839527 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:43.843762 13879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41479
I20260928 09:01:43.843817 13946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41479 every 8 connection(s)
I20260928 09:01:43.844170 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:01:43.844465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 13879
I20260928 09:01:43.844594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585920139418-392-0/minicluster-data/master-2/wal/instance
I20260928 09:01:43.844830 392 external_mini_cluster.cc:1428] Setting key 9c8be05202b25df92b986b7bc0479794
I20260928 09:01:43.845208 13947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:43.847476 13947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:43.856555 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.845352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.0.98.60:41479)
0928 09:01:43.845548 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.845557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.845591 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:01:43.846314 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.846323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.846387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.846544 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.847146 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.848412 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.849418 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.849952 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.850080 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.853046 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.853052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.853052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.853058 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.854951 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.855345 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.855348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.855348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.855422 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.855635 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.855636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.855637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.855939 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.856070 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.856201 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.856301 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260928 09:01:43.856824 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.848185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.62:39955)
0928 09:01:43.848233 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.848235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.848374 (+ 139us) server_negotiation.cc:407] Connection header received
0928 09:01:43.848419 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.848420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.848432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.848493 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.849084 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.849666 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.850668 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.851014 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.851096 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.853795 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.853797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.853797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.853804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.855493 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.855898 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.855899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.855899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.855949 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.856176 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.856177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.856177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.856330 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.856404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.856602 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.856719 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 09:01:43.856842 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.848111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.0.1:47332)
0928 09:01:43.848259 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.848302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.848373 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.848524 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.848526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.848916 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.848990 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.848993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.849765 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.849766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.850565 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.850569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.851118 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.851119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.851174 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.851640 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.851648 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.853672 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.855567 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.855571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.855573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.855823 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.856040 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.856041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.856041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.856104 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.856398 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.856405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.856462 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.856573 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.856728 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260928 09:01:43.857792 13947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:01:43.859596 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } attempt: 2
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:01:43.866168 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.858153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.0.98.61:43903)
0928 09:01:43.858192 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.858194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.858228 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:01:43.858276 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.858277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.858284 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.858323 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.858626 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.859153 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.860021 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.860383 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.860448 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.862884 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.862887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.862887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.862895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.864583 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.864986 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.864988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.864988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.865316 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.865581 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.865583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.865583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.865733 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.865796 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.865989 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.866066 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:01:43.866168 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.858080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43903 (local address 127.0.0.1:59300)
0928 09:01:43.858135 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.858146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.858185 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.858354 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.858355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.858509 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.858539 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.858541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.859260 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.859261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.859913 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.859916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.860455 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.860456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.860511 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.861161 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.861166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.862771 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.864696 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.864697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.864697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.864906 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.865406 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.865407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.865407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.865493 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.865802 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.865803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.865853 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.865956 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.866056 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 09:01:43.866640 13947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:01:43.867226 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.857535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.98.62:39955)
0928 09:01:43.857582 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.857585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.857592 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:01:43.857610 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.857611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.857620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.857675 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.858404 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.859067 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.859982 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.860282 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.860351 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.863026 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.863027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.863028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.863033 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.865067 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.865450 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.865451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.865451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.865503 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.865740 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.865741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.865741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.865879 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.867022 (+ 1143us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.867069 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.867146 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 09:01:43.869275 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.859925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.98.60:41479)
0928 09:01:43.860008 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.860009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.860016 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:01:43.860031 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.860031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.860039 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.860382 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.860407 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.861157 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.862163 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.862394 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.862489 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.864851 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.864852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.864852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.864859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.868188 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.868523 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.868524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.868524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.868576 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.868799 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.868800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.868801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.868908 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.868982 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.869124 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.869192 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 09:01:43.869292 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.859831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43570)
0928 09:01:43.859887 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.859898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.859941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.860120 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.860120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.860285 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.860318 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.860319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.861261 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.861262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.862052 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.862054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.862490 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.862491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.862554 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.863161 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.863166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.864763 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.868265 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.868266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.868266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.868435 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.868664 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.868665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.868666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.868722 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.868975 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.868976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.869017 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.869101 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.869190 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:01:43.871107 13797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c: Bootstrap starting.
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:01:43.871901 13797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:43.872210 13797 log.cc:824] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c: Log is configured to *not* fsync() on all Append() calls
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:01:43.873209 13797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c: No bootstrap required, opened a new log
I20260928 09:01:43.874636 13797 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.874743 13797 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:43.874768 13797 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54db4ddbd7741f28f4838af1a260b5c, State: Initialized, Role: FOLLOWER
I20260928 09:01:43.874897 13797 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.875233 13797 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:43.876753 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.867933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.98.61:43903)
0928 09:01:43.867966 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.867968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.867977 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:01:43.868063 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.868064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.868073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.868132 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.868756 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.869508 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.870394 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.870710 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.870814 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.873371 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.873372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.873372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.873379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.875057 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.875512 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.875513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.875513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.875569 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.875810 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.875811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.875811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.875971 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.876477 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.876524 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.876610 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:01:43.876896 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.866920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43586)
0928 09:01:43.866955 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.866964 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.867002 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.867289 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.867290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.867474 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.867510 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.867512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.869020 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.869021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.869828 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.869830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.870333 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.870334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.870371 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.870792 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.870798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.872110 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.875638 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.875639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.875639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.875898 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.876163 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.876164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.876164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.876234 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.876548 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.876549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.876596 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.876704 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.876803 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 09:01:43.877461 13965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:01:43.877518 13965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:01:43.877667 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.866977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.0.98.60:41479)
0928 09:01:43.867152 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.867154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.867160 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:01:43.867175 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.867175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.867183 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.867286 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.867611 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.868940 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.869910 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.870224 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.870288 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.872209 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.872211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.872211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.872220 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.875540 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.876012 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.876013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.876014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.876071 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.876298 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.876298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.876299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.876476 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.876544 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.877459 (+ 915us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.877546 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260928 09:01:43.875183 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.878455 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:01:43.878393 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.879755 13729 master_runner.cc:186] Error getting master registration for 127.0.98.61:43903: OK, instance_id { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" instance_seqno: 1790586103683180 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:01:43.882582 13729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:01:43.884027 13947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: Bootstrap starting.
I20260928 09:01:43.884930 13947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:43.885279 13947 log.cc:824] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:43.887272 13947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: No bootstrap required, opened a new log
I20260928 09:01:43.890072 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.880042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.0.98.61:43903)
0928 09:01:43.880075 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.880077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.880183 (+ 106us) server_negotiation.cc:407] Connection header received
0928 09:01:43.880269 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.880270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.880281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.880341 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.881102 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.881885 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.883273 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.883585 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.883633 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.886441 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.886442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.886443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.886451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.888315 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.888755 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.888757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.888757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.888815 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.889026 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.889027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.889027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.889169 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.889279 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.889823 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.889908 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 09:01:43.890611 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.879814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.0.98.62:39955)
0928 09:01:43.879855 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.879857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.880359 (+ 502us) server_negotiation.cc:407] Connection header received
0928 09:01:43.880462 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.880463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.880472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.880500 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.881212 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.881934 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.883126 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.883426 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.883469 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.885113 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.885114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.885115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.885121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.886830 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.889018 (+ 2188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.889019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.889020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.889079 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.889283 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.889284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.889285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.889403 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.889497 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.890441 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.890517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:01:43.890959 13947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.891106 13947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:43.891141 13947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7c6dde80ad4ce9972a913529631c65, State: Initialized, Role: FOLLOWER
I20260928 09:01:43.891283 13947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.891575 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.891662 13947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:43.892696 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.894075 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.882667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.0.98.60:41479)
0928 09:01:43.882707 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.882709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.885688 (+ 2979us) server_negotiation.cc:407] Connection header received
0928 09:01:43.885820 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.885821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.885835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.885870 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.886532 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.887449 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.888601 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.888990 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.889035 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.890534 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.890536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.890536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.890544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.892464 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.892996 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.892997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.892998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.893064 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.893318 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.893319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.893319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.893465 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.893580 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.893858 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.893976 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:01:43.894374 13986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:01:43.894438 13986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:01:43.895987 13879 master_runner.cc:186] Error getting master registration for 127.0.98.61:43903: OK, instance_id { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" instance_seqno: 1790586103683180 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:01:43.897864 13879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:01:43.899983 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:43.900084 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.900321 13954 leader_election.cc:288] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8b434a80184f7a83d5e457fc3cc89d (127.0.98.61:43903), 4a7c6dde80ad4ce9972a913529631c65 (127.0.98.60:41479)
I20260928 09:01:43.900579 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c54db4ddbd7741f28f4838af1a260b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7c6dde80ad4ce9972a913529631c65" is_pre_election: true
I20260928 09:01:43.900777 13921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c54db4ddbd7741f28f4838af1a260b5c in term 0.
I20260928 09:01:43.900985 13747 leader_election.cc:302] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7c6dde80ad4ce9972a913529631c65, c54db4ddbd7741f28f4838af1a260b5c; no voters:
I20260928 09:01:43.901114 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:01:43.901189 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:01:43.901214 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:43.902140 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.902279 13954 leader_election.cc:288] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [CANDIDATE]: Term 1 election: Requested vote from peers 4f8b434a80184f7a83d5e457fc3cc89d (127.0.98.61:43903), 4a7c6dde80ad4ce9972a913529631c65 (127.0.98.60:41479)
I20260928 09:01:43.902441 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c54db4ddbd7741f28f4838af1a260b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7c6dde80ad4ce9972a913529631c65"
I20260928 09:01:43.902513 13921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:43.903942 13921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c54db4ddbd7741f28f4838af1a260b5c in term 1.
I20260928 09:01:43.904188 13747 leader_election.cc:302] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7c6dde80ad4ce9972a913529631c65, c54db4ddbd7741f28f4838af1a260b5c; no voters:
I20260928 09:01:43.904292 13954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:43.904521 13954 raft_consensus.cc:686] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [term 1 LEADER]: Becoming Leader. State: Replica: c54db4ddbd7741f28f4838af1a260b5c, State: Running, Role: LEADER
I20260928 09:01:43.904650 13954 consensus_queue.cc:237] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.905122 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54db4ddbd7741f28f4838af1a260b5c. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.905202 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:43.912775 13990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:43.913255 13990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:01:43.915066 13921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [term 1 FOLLOWER]: Refusing update from remote peer c54db4ddbd7741f28f4838af1a260b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:43.915429 13988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:43.916769 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54db4ddbd7741f28f4838af1a260b5c. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.916857 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.918597 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.918686 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:43.918740 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.918825 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.918816 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } attempt: 3
I20260928 09:01:43.919039 13991 mvcc.cc:205] Tried to move back new op lower bound from 7334240681632972800 to 7334240681593827328. Current Snapshot: MvccSnapshot[applied={T|T < 7334240681632972800}]
I20260928 09:01:43.919145 13992 mvcc.cc:205] Tried to move back new op lower bound from 7334240681632972800 to 7334240681593827328. Current Snapshot: MvccSnapshot[applied={T|T < 7334240681632972800}]
Sep 28 09:01:43 dist-test-slave-286n krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:01:43.927915 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.919083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41479 (local address 127.0.0.1:43590)
0928 09:01:43.919170 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:43.919182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:43.919221 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:43.919690 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:43.919691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:43.919896 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:43.919945 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.919947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.920982 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.920983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.921835 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:43.921837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.922438 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.922439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:43.922512 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.923074 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:43.923094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:43.924731 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:43.926684 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.926685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.926685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.926904 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.927164 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:43.927164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:43.927165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:43.927237 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:43.927585 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:43.927586 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:43.927647 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 09:01:43.927710 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 09:01:43.927808 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 09:01:43.928354 13990 catalog_manager.cc:1347] Generated new cluster ID: d4ec1df6e77147c5b4025b54277f0f63
I20260928 09:01:43.928419 13990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:01:43.933115 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:43.919304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.0.98.60:41479)
0928 09:01:43.919342 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:43.919344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:43.919351 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:01:43.919372 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:43.919373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:43.919382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:43.919428 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:43.920058 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.920861 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.921952 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:43.922279 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:43.922336 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:43.924833 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:43.924835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:43.924835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:43.924842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:43.926570 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.927004 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.927005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.927005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.927077 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:43.927330 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:43.927331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:43.927331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:43.927478 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:43.927537 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:43.932910 (+ 5373us) server_negotiation.cc:299] Negotiation successful
0928 09:01:43.933012 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:01:43.934743 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: Bootstrap starting.
I20260928 09:01:43.935462 13872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:43.935647 13990 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:01:43.935758 13872 log.cc:824] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:43.936537 13990 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:01:43.936787 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: No bootstrap required, opened a new log
I20260928 09:01:43.938823 13872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.938977 13872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:43.939018 13872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8b434a80184f7a83d5e457fc3cc89d, State: Initialized, Role: FOLLOWER
I20260928 09:01:43.939136 13872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } }
I20260928 09:01:43.939456 13872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:43.940128 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c54db4ddbd7741f28f4838af1a260b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" is_pre_election: true
I20260928 09:01:43.940976 13844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:43.941910 13844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [term 1 FOLLOWER]: Refusing update from remote peer c54db4ddbd7741f28f4838af1a260b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:43.940282 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c54db4ddbd7741f28f4838af1a260b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8b434a80184f7a83d5e457fc3cc89d"
I20260928 09:01:43.943635 13845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c54db4ddbd7741f28f4838af1a260b5c in term 1.
I20260928 09:01:43.944157 13995 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 09:01:43.947006 14013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:01:43.947067 14013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:01:43.947206 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.947293 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.947356 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c54db4ddbd7741f28f4838af1a260b5c. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.947412 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.947444 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.947492 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:01:43.948102 14004 mvcc.cc:205] Tried to move back new op lower bound from 7334240681712021504 to 7334240681593827328. Current Snapshot: MvccSnapshot[applied={T|T < 7334240681632972800}]
I20260928 09:01:43.949347 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c54db4ddbd7741f28f4838af1a260b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c54db4ddbd7741f28f4838af1a260b5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } peers { permanent_uuid: "4f8b434a80184f7a83d5e457fc3cc89d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43903 } } peers { permanent_uuid: "4a7c6dde80ad4ce9972a913529631c65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41479 } } }
I20260928 09:01:43.949452 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:43.949978 13990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c54db4ddbd7741f28f4838af1a260b5c: Generated new TSK 0
I20260928 09:01:43.950169 13990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:01:44.894909 13986 catalog_manager.cc:1259] Loaded cluster ID: d4ec1df6e77147c5b4025b54277f0f63
I20260928 09:01:44.895030 13986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: loading cluster ID for follower catalog manager: success
I20260928 09:01:44.895849 13986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: acquiring CA information for follower catalog manager: success
I20260928 09:01:44.896159 13986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a7c6dde80ad4ce9972a913529631c65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:01:44.947503 14013 catalog_manager.cc:1259] Loaded cluster ID: d4ec1df6e77147c5b4025b54277f0f63
I20260928 09:01:44.947616 14013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: loading cluster ID for follower catalog manager: success
I20260928 09:01:44.948518 14013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: acquiring CA information for follower catalog manager: success
I20260928 09:01:44.948858 14013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f8b434a80184f7a83d5e457fc3cc89d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:01:45.009159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13729
I20260928 09:01:45.013950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13804
I20260928 09:01:45.018759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 13879
2026-09-28T09:01:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2828 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:01:45 dist-test-slave-286n krb5kdc[14026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:01:45 dist-test-slave-286n krb5kdc[14026](info): set up 2 sockets
Sep 28 09:01:45 dist-test-slave-286n krb5kdc[14026](info): commencing operation
krb5kdc: starting...
W20260928 09:01:46.133177 392 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:01:46 dist-test-slave-286n krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:01:46.252815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41807
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:46.335346 14039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:46.335644 14039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:46.335690 14039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:46.337108 14039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:46.337189 14039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:46.337216 14039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:46.337241 14039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:46.337292 14039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:46.339027 14039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:46.339329 14039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:46.339569 14039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:46.344724 14047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:46.344728 14045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:46.344727 14044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:46.344779 14039 server_base.cc:1034] running on GCE node
I20260928 09:01:46.345265 14039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:46.345558 14039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:46.346735 14039 hybrid_clock.cc:648] HybridClock initialized: now 1790586106346715 us; error 29 us; skew 500 ppm
Sep 28 09:01:46 dist-test-slave-286n krb5kdc[14026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:46.349438 14039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:46.349937 14039 webserver.cc:460] Webserver started at http://127.0.98.62:45929/ using document root <none> and password file <none>
I20260928 09:01:46.350198 14039 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:46.350270 14039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:46.350419 14039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:46.351506 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "2962fc44ec414b829f138401ab008f6e"
format_stamp: "Formatted at 2026-09-28 09:01:46 on dist-test-slave-286n"
server_key: "3c6c5c93467f0d3c0aed040d7492a8a7"
server_key_iv: "cdda34be3d7bd1b429cc5ef77b842a65"
server_key_version: "encryptionkey@0"
I20260928 09:01:46.351886 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "2962fc44ec414b829f138401ab008f6e"
format_stamp: "Formatted at 2026-09-28 09:01:46 on dist-test-slave-286n"
server_key: "3c6c5c93467f0d3c0aed040d7492a8a7"
server_key_iv: "cdda34be3d7bd1b429cc5ef77b842a65"
server_key_version: "encryptionkey@0"
I20260928 09:01:46.353291 14039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:01:46.354034 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:46.354206 14039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:01:46.354310 14039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "2962fc44ec414b829f138401ab008f6e"
format_stamp: "Formatted at 2026-09-28 09:01:46 on dist-test-slave-286n"
server_key: "3c6c5c93467f0d3c0aed040d7492a8a7"
server_key_iv: "cdda34be3d7bd1b429cc5ef77b842a65"
server_key_version: "encryptionkey@0"
I20260928 09:01:46.354385 14039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:46.371771 14039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:46.374150 14039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:46.374320 14039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:46.378554 14039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41807
I20260928 09:01:46.378592 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41807 every 8 connection(s)
I20260928 09:01:46.378957 14039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:46.379825 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:46.382158 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e: Bootstrap starting.
I20260928 09:01:46.382706 14107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:46.382936 14107 log.cc:824] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:46.383611 14107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e: No bootstrap required, opened a new log
I20260928 09:01:46.385244 14107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2962fc44ec414b829f138401ab008f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41807 } }
I20260928 09:01:46.385382 14107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:46.385406 14107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2962fc44ec414b829f138401ab008f6e, State: Initialized, Role: FOLLOWER
I20260928 09:01:46.385517 14107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2962fc44ec414b829f138401ab008f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41807 } }
I20260928 09:01:46.385596 14107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:01:46.385643 14107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:01:46.385689 14107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:46.386226 14107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2962fc44ec414b829f138401ab008f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41807 } }
I20260928 09:01:46.386339 14107 leader_election.cc:302] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2962fc44ec414b829f138401ab008f6e; no voters:
I20260928 09:01:46.386530 14107 leader_election.cc:288] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:01:46.386574 14110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:46.386761 14110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [term 1 LEADER]: Becoming Leader. State: Replica: 2962fc44ec414b829f138401ab008f6e, State: Running, Role: LEADER
I20260928 09:01:46.386794 14107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:46.386857 14110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2962fc44ec414b829f138401ab008f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41807 } }
I20260928 09:01:46.387269 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2962fc44ec414b829f138401ab008f6e. Latest consensus state: current_term: 1 leader_uuid: "2962fc44ec414b829f138401ab008f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2962fc44ec414b829f138401ab008f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41807 } } }
I20260928 09:01:46.387399 14111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2962fc44ec414b829f138401ab008f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2962fc44ec414b829f138401ab008f6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41807 } } }
I20260928 09:01:46.387497 14111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:46.387411 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:46.387820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14039
I20260928 09:01:46.387919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:01:46.387934 14119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:46.388123 392 external_mini_cluster.cc:1428] Setting key 164676b96c55271620c72e275eb8828d
I20260928 09:01:46.388456 14119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:01:46.390286 14119 catalog_manager.cc:1347] Generated new cluster ID: 45aad684d9224e2a94b300b30078f337
I20260928 09:01:46.390340 14119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:01:46 dist-test-slave-286n krb5kdc[14026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:46.401332 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:46.388620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.98.62:41807)
0928 09:01:46.389360 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:46.389369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:46.389403 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:01:46.389520 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:46.389527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:46.389584 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:46.389726 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:46.393148 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:46.394063 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:46.394864 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:46.395332 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:46.395451 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:46.397830 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:46.397835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:46.397836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:46.397848 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:46.399547 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:46.399918 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:46.399923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:46.399928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:46.400016 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:46.400384 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:46.400385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:46.400386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:46.400693 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:46.400881 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:46.400961 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 09:01:46.401105 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":75,"threads_started":1}
I20260928 09:01:46.402427 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:01:46.410660 14119 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:01:46.411405 14119 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:01:46.415375 14119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2962fc44ec414b829f138401ab008f6e: Generated new TSK 0
I20260928 09:01:46.415568 14119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:01:46.456354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14039
2026-09-28T09:01:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1436 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:01:46 dist-test-slave-286n krb5kdc[14134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:01:46 dist-test-slave-286n krb5kdc[14134](info): set up 2 sockets
Sep 28 09:01:46 dist-test-slave-286n krb5kdc[14134](info): commencing operation
krb5kdc: starting...
W20260928 09:01:47.492842 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:01:47 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:01:47.617488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:47.700428 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:47.700662 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:47.700685 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:47.701879 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:47.701920 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:47.701933 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:47.701946 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:47.701998 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:47.703681 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:47.703938 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:47.704170 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:47.709122 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:47.709185 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:47.709187 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:47.709292 14147 server_base.cc:1034] running on GCE node
I20260928 09:01:47.709640 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:47.709921 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:47.711092 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790586107711073 us; error 44 us; skew 500 ppm
Sep 28 09:01:47 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:47.713605 14147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:47.714033 14147 webserver.cc:460] Webserver started at http://127.0.98.62:41493/ using document root <none> and password file <none>
I20260928 09:01:47.714260 14147 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:47.714327 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:47.714473 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:47.715598 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "7269e2e57b9f48f3be73e801cc7e2b35"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "d2311a21551c9ac777f0bf9a36084e9a"
server_key_iv: "00538fb046cbc958cb1db8f186ee34d7"
server_key_version: "encryptionkey@0"
I20260928 09:01:47.715958 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "7269e2e57b9f48f3be73e801cc7e2b35"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "d2311a21551c9ac777f0bf9a36084e9a"
server_key_iv: "00538fb046cbc958cb1db8f186ee34d7"
server_key_version: "encryptionkey@0"
I20260928 09:01:47.717438 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:01:47.718236 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:47.718436 14147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:01:47.718540 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "7269e2e57b9f48f3be73e801cc7e2b35"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "d2311a21551c9ac777f0bf9a36084e9a"
server_key_iv: "00538fb046cbc958cb1db8f186ee34d7"
server_key_version: "encryptionkey@0"
I20260928 09:01:47.718613 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:47.741441 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:47.743553 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:47.743673 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:47.747530 14147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42083
I20260928 09:01:47.747586 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42083 every 8 connection(s)
I20260928 09:01:47.747939 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:47.749042 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:47.751294 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap starting.
I20260928 09:01:47.751873 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:47.752167 14215 log.cc:824] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:47.752842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14147
I20260928 09:01:47.752926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:01:47.752981 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: No bootstrap required, opened a new log
I20260928 09:01:47.753132 392 external_mini_cluster.cc:1428] Setting key f81b300b7f36b0ed5dda95b01c2264b0
I20260928 09:01:47.755385 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:47.755565 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:47.755610 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7269e2e57b9f48f3be73e801cc7e2b35, State: Initialized, Role: FOLLOWER
I20260928 09:01:47.755760 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:47.755837 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:01:47.755867 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:01:47.755911 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:47.756623 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:47.756754 14215 leader_election.cc:302] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7269e2e57b9f48f3be73e801cc7e2b35; no voters:
I20260928 09:01:47.756896 14215 leader_election.cc:288] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:01:47.756953 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:47.757144 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:47.757143 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 LEADER]: Becoming Leader. State: Replica: 7269e2e57b9f48f3be73e801cc7e2b35, State: Running, Role: LEADER
I20260928 09:01:47.757426 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:47.757922 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7269e2e57b9f48f3be73e801cc7e2b35. Latest consensus state: current_term: 1 leader_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 09:01:47.758009 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:47.758436 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 09:01:47.758498 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: This master's current role is: LEADER
W20260928 09:01:47.759166 14235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:01:47.759220 14235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:01:47.759268 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:47.759402 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:01:47.760929 14229 catalog_manager.cc:1347] Generated new cluster ID: 486eb4770173499dbe026e569773395b
I20260928 09:01:47.760982 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:01:47 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:47.765015 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:47.753658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.0.98.62:42083)
0928 09:01:47.753896 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:47.753906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:47.753938 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:01:47.754546 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:47.754556 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:47.754620 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:47.754782 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:47.755381 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:47.757078 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:47.758295 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:47.758950 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:47.759075 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:47.761711 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:47.761715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:47.761715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:47.761728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:47.763436 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:47.763833 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:47.763836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:47.763837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:47.763926 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:47.764142 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:47.764142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:47.764143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:47.764378 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:47.764559 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:47.764630 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 09:01:47.764796 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":121,"threads_started":1}
I20260928 09:01:47.784870 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:01:47.785476 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:01:47.790174 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Generated new TSK 0
I20260928 09:01:47.790410 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:01:47.824069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:47.925441 14241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:47.925645 14241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:47.925669 14241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:47.926823 14241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:47.926867 14241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:47.926894 14241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:01:47.928347 14241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:47.928588 14241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:47.928855 14241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:47.933928 14249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:47.933997 14247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:47.934068 14241 server_base.cc:1034] running on GCE node
W20260928 09:01:47.933960 14246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:47.934438 14241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:47.934638 14241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:47.935775 14241 hybrid_clock.cc:648] HybridClock initialized: now 1790586107935757 us; error 33 us; skew 500 ppm
Sep 28 09:01:47 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:47.938313 14241 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:01:47.938745 14241 webserver.cc:460] Webserver started at http://127.0.98.1:37235/ using document root <none> and password file <none>
I20260928 09:01:47.938980 14241 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:47.939026 14241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:47.939199 14241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:47.940335 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "83440db9bf8a4040b1a2335a65af3813"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "51fd2a4b8ba6aebbebe8de8c5fc0dbc5"
server_key_iv: "1b8dcb5471d313f3f87f6d3d36737eb5"
server_key_version: "encryptionkey@0"
I20260928 09:01:47.940754 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83440db9bf8a4040b1a2335a65af3813"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "51fd2a4b8ba6aebbebe8de8c5fc0dbc5"
server_key_iv: "1b8dcb5471d313f3f87f6d3d36737eb5"
server_key_version: "encryptionkey@0"
I20260928 09:01:47.942072 14241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:47.942868 14256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:47.943058 14241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:01:47.943128 14241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "83440db9bf8a4040b1a2335a65af3813"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "51fd2a4b8ba6aebbebe8de8c5fc0dbc5"
server_key_iv: "1b8dcb5471d313f3f87f6d3d36737eb5"
server_key_version: "encryptionkey@0"
I20260928 09:01:47.943264 14241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:47.950939 14241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:47.952847 14241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:47.952996 14241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:47.953212 14241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:47.953523 14241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:47.953576 14241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:47.953617 14241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:47.953645 14241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:47.959537 14241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36993
I20260928 09:01:47.959610 14369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36993 every 8 connection(s)
I20260928 09:01:47.959954 14241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:01:47 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:47.969188 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:47.960806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39407 (local address 127.0.98.62:42083)
0928 09:01:47.960866 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:47.960868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:47.961073 (+ 205us) server_negotiation.cc:407] Connection header received
0928 09:01:47.961632 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:47.961633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:47.961644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:47.961698 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:47.962608 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:47.963116 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:47.964098 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:47.964300 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:47.964351 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:47.966517 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:47.966518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:47.966519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:47.966525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:47.967901 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:47.968290 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:47.968291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:47.968291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:47.968341 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:47.968583 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:47.968583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:47.968584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:47.968754 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:47.968808 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:47.969052 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 09:01:47.969120 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 09:01:47.969359 14372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:47.960802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:39407)
0928 09:01:47.960963 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:47.960999 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:47.961522 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:47.961754 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:47.961758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:47.962329 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:47.962528 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:47.962533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:47.963240 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:47.963240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:47.964022 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:47.964024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:47.964374 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:47.964375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:47.964484 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:47.964985 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:47.964993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:47.966425 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:47.967979 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:47.967986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:47.967987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:47.968213 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:47.968415 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:47.968415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:47.968416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:47.968526 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:47.968814 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:47.968819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:47.968939 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 09:01:47.969026 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 09:01:47.969148 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260928 09:01:47.969939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14241
I20260928 09:01:47.969991 14370 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 09:01:47.970053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:01:47.970121 14370 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:47.970278 392 external_mini_cluster.cc:1428] Setting key 7bd70061a18c8491c1c2f4a675eaf1ef
I20260928 09:01:47.970331 14370 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 09:01:47.971009 14178 ts_manager.cc:194] Registered new tserver with Master: 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993)
I20260928 09:01:47.971621 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39407
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:01:47.997035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:48.080242 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:48.080459 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:48.080484 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:48.081724 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:48.081774 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:48.081804 14375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:01:48.083240 14375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:48.083441 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:48.083632 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:48.088863 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.088814 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.088838 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.089149 14375 server_base.cc:1034] running on GCE node
I20260928 09:01:48.089375 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:48.089656 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:48.090817 14375 hybrid_clock.cc:648] HybridClock initialized: now 1790586108090797 us; error 43 us; skew 500 ppm
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:48.100559 14375 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:01:48.101052 14375 webserver.cc:460] Webserver started at http://127.0.98.2:39699/ using document root <none> and password file <none>
I20260928 09:01:48.101264 14375 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:48.101308 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:48.101404 14375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:48.102535 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f23c0d3d1664e56beca6d00861a6477"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "90119dc480417d9dc4e6cbbafab8debe"
server_key_iv: "c06a219844c036870fb3a69ad6795468"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.102864 14375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f23c0d3d1664e56beca6d00861a6477"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "90119dc480417d9dc4e6cbbafab8debe"
server_key_iv: "c06a219844c036870fb3a69ad6795468"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.104213 14375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:01:48.105132 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.105351 14375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:01:48.105458 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "6f23c0d3d1664e56beca6d00861a6477"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "90119dc480417d9dc4e6cbbafab8debe"
server_key_iv: "c06a219844c036870fb3a69ad6795468"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.105533 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:48.112815 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:48.114812 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:48.114985 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:48.115221 14375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:48.115521 14375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:48.115576 14375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.115615 14375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:48.115644 14375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.121620 14375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35735
I20260928 09:01:48.121661 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35735 every 8 connection(s)
I20260928 09:01:48.122045 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:48.131956 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.122802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38163 (local address 127.0.98.62:42083)
0928 09:01:48.122875 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.122877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.123064 (+ 187us) server_negotiation.cc:407] Connection header received
0928 09:01:48.123533 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.123534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.123544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.123611 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.124541 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.125229 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.126225 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.126441 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.126506 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.128984 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.128986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.128986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.128993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.130674 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.131081 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.131082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.131083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.131145 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.131374 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.131374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.131375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.131492 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.131551 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.131815 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.131885 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:01:48.132126 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.122784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:38163)
0928 09:01:48.122937 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.122978 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.123452 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.123664 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.123667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.124235 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:48.124428 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.124432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.125332 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.125333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.126132 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.126134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.126560 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.126562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.126685 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.127255 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:48.127264 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:48.128839 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:48.130758 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.130765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.130767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.130991 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.131214 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.131214 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.131215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.131312 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.131551 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:48.131555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:48.131672 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.131775 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.131898 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 09:01:48.132328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14375
I20260928 09:01:48.132457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:01:48.132714 392 external_mini_cluster.cc:1428] Setting key ba3bb7eeaa6b57b7eecce190d092f494
I20260928 09:01:48.132779 14504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 09:01:48.132932 14504 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:48.133147 14504 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 09:01:48.133577 14179 ts_manager.cc:194] Registered new tserver with Master: 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735)
I20260928 09:01:48.133985 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38163
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:01:48.160295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:48.254002 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:48.254256 14509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:48.254280 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:48.255506 14509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:48.255551 14509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:48.255609 14509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:01:48.257182 14509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:48.257437 14509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:48.257707 14509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:48.263055 14514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.263064 14515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.263053 14517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.263139 14509 server_base.cc:1034] running on GCE node
I20260928 09:01:48.263540 14509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:48.263780 14509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:48.264921 14509 hybrid_clock.cc:648] HybridClock initialized: now 1790586108264911 us; error 32 us; skew 500 ppm
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:48.267431 14509 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:01:48.267875 14509 webserver.cc:460] Webserver started at http://127.0.98.3:36721/ using document root <none> and password file <none>
I20260928 09:01:48.268093 14509 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:48.268137 14509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:48.268307 14509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:01:48.269397 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "7962d529ebcc42e38d4db7542a9e17d7"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "1f7e97729d0daaea14be8ae9946a7654"
server_key_iv: "0402990bb9da6a6d722c9db95d9c44d5"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.269747 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7962d529ebcc42e38d4db7542a9e17d7"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "1f7e97729d0daaea14be8ae9946a7654"
server_key_iv: "0402990bb9da6a6d722c9db95d9c44d5"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.271106 14509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:01:48.271864 14524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.272065 14509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:48.272164 14509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "7962d529ebcc42e38d4db7542a9e17d7"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "1f7e97729d0daaea14be8ae9946a7654"
server_key_iv: "0402990bb9da6a6d722c9db95d9c44d5"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.272243 14509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:48.295158 14509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:48.297402 14509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:48.297582 14509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:48.297851 14509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:48.298328 14509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:48.298403 14509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.298453 14509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:48.298520 14509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.305193 14509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33381
I20260928 09:01:48.305235 14637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33381 every 8 connection(s)
I20260928 09:01:48.305612 14509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:48.315361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14509
I20260928 09:01:48.315481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:01:48.315596 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.306435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39549 (local address 127.0.98.62:42083)
0928 09:01:48.306489 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.306491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.306708 (+ 217us) server_negotiation.cc:407] Connection header received
0928 09:01:48.307282 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.307283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.307293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.307348 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.308477 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.309095 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.310268 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.310460 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.310527 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.312705 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.312707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.312707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.312715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.314186 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.314579 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.314581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.314581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.314641 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.314889 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.314890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.314890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.315012 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.315069 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.315380 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.315483 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:01:48.315666 392 external_mini_cluster.cc:1428] Setting key 3554bd58b72780c03e94a0c3be405c7e
I20260928 09:01:48.315805 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.306422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:39549)
0928 09:01:48.306573 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.306613 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.307180 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.307408 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.307411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.308066 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:48.308370 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.308376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.309318 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.309320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.310160 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.310162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.310544 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.310545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.310646 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.311156 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:48.311165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:48.312544 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:48.314260 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.314267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.314269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.314500 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.314703 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.314703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.314704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.314825 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.315067 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:48.315071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:48.315232 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.315358 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.315549 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260928 09:01:48.316366 14638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 09:01:48.316491 14638 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:48.316761 14638 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 09:01:48.317107 14179 ts_manager.cc:194] Registered new tserver with Master: 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
I20260928 09:01:48.317510 14179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39549
I20260928 09:01:48.327713 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:01:48.335258 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.329187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.0.98.62:42083)
0928 09:01:48.329266 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.329269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.329364 (+ 95us) server_negotiation.cc:407] Connection header received
0928 09:01:48.329442 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.329443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.329455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.329525 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.330113 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.330622 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.331353 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.331548 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.331622 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.332911 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.332913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.332913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.332921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.334060 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.334407 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.334408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.334408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.334470 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.334661 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.334662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.334662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.334823 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.334944 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.335061 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.335148 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:01:48.336946 14179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:01:48.338513 14179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:01:48.344993 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.340748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36993 (local address 127.0.0.1:32864)
0928 09:01:48.340883 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.340916 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.340995 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.341209 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.341212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.341214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.341289 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.341290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.342608 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.342609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.343779 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.343781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.344724 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.344725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.344805 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.344827 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.344896 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.344934 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260928 09:01:48.345053 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.340866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.98.1:36993)
0928 09:01:48.341048 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.341054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.341064 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:01:48.341102 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.341104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.341118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.341207 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.341389 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.342509 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.343870 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.344621 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.344724 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.344783 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.344926 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.344969 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260928 09:01:48.345348 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.340989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33381 (local address 127.0.0.1:47344)
0928 09:01:48.341113 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.341123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.341168 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.341873 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.341874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.341877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.341932 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.341934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.343019 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.343021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.344038 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.344040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.345111 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.345112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.345199 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.345203 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.345254 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.345290 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260928 09:01:48.345408 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.341353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.0.98.3:33381)
0928 09:01:48.341640 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.341645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.341655 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:01:48.341694 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.341696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.341712 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.341794 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.342014 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.342927 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.344113 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.344617 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.344728 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.344788 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.345272 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.345318 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260928 09:01:48.345913 14304 tablet_service.cc:1467] Processing CreateTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 (DEFAULT_TABLE table=test-table [id=72e0daed854a4fb18f31b198a04c9405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:48.346256 14304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6743ec7f3d70426298bf2d5d548f7120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:48.346143 14572 tablet_service.cc:1467] Processing CreateTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 (DEFAULT_TABLE table=test-table [id=72e0daed854a4fb18f31b198a04c9405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:48.346484 14572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6743ec7f3d70426298bf2d5d548f7120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:48.346706 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.341166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35735 (local address 127.0.0.1:54702)
0928 09:01:48.341450 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.341460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.341508 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.341881 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.341882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.341883 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.341932 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.341935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.343995 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.343997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.345387 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.345390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.346469 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.346470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.346537 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.346541 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.346595 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.346639 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260928 09:01:48.346764 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.341464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.98.2:35735)
0928 09:01:48.341640 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.341645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.341655 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:01:48.341695 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.341696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.341712 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.341803 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.343023 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.343893 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.345484 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.346281 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.346353 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.346430 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.346629 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.346679 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260928 09:01:48.349337 14655 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Bootstrap starting.
I20260928 09:01:48.349407 14656 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Bootstrap starting.
I20260928 09:01:48.349337 14438 tablet_service.cc:1467] Processing CreateTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 (DEFAULT_TABLE table=test-table [id=72e0daed854a4fb18f31b198a04c9405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:48.349649 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6743ec7f3d70426298bf2d5d548f7120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:48.350100 14656 tablet_bootstrap.cc:654] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:48.350100 14655 tablet_bootstrap.cc:654] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:48.350675 14656 log.cc:824] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.350675 14655 log.cc:824] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.351430 14655 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: No bootstrap required, opened a new log
I20260928 09:01:48.351495 14655 ts_tablet_manager.cc:1397] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:01:48.351737 14656 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: No bootstrap required, opened a new log
I20260928 09:01:48.351791 14656 ts_tablet_manager.cc:1397] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:01:48.352284 14659 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Bootstrap starting.
I20260928 09:01:48.352913 14659 tablet_bootstrap.cc:654] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:48.353374 14659 log.cc:824] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.353747 14656 raft_consensus.cc:348] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.353747 14655 raft_consensus.cc:348] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.353941 14656 raft_consensus.cc:374] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:48.353941 14655 raft_consensus.cc:374] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:48.353991 14655 raft_consensus.cc:729] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83440db9bf8a4040b1a2335a65af3813, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.354022 14656 raft_consensus.cc:729] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7962d529ebcc42e38d4db7542a9e17d7, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.354045 14659 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: No bootstrap required, opened a new log
I20260928 09:01:48.354099 14659 ts_tablet_manager.cc:1397] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 09:01:48.354161 14656 consensus_queue.cc:260] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.354164 14655 consensus_queue.cc:260] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.354449 14656 ts_tablet_manager.cc:1428] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:01:48.354449 14655 ts_tablet_manager.cc:1428] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:01:48.354569 14638 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 09:01:48.354887 14370 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 09:01:48.356350 14659 raft_consensus.cc:348] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.356539 14659 raft_consensus.cc:374] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:48.356511 14661 raft_consensus.cc:482] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:48.356566 14659 raft_consensus.cc:729] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f23c0d3d1664e56beca6d00861a6477, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.356616 14661 raft_consensus.cc:504] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.356693 14659 consensus_queue.cc:260] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.356956 14659 ts_tablet_manager.cc:1428] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:01:48.356966 14504 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 09:01:48.357009 14661 leader_election.cc:288] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993), 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735)
I20260928 09:01:48.360409 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.357202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46237 (local address 127.0.98.1:36993)
0928 09:01:48.357412 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.357414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.357424 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:01:48.357449 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.357450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.357460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.357510 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.357744 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.358377 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.359561 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.360036 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.360089 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.360098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.360264 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.360309 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260928 09:01:48.360409 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.357158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36993 (local address 127.0.98.3:46237)
0928 09:01:48.357245 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.357257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.357300 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.357617 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.357618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.357620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.357683 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.357685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.358447 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.358448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.359476 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.359479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.360108 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.360109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.360196 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.360200 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.360250 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.360307 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 09:01:48.360430 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.357613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55825 (local address 127.0.98.2:35735)
0928 09:01:48.357668 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.357670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.357678 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:01:48.357698 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.357699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.357707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.357757 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.357974 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.358612 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.359620 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.360104 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.360173 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.360182 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.360324 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.360370 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:01:48.360486 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.357269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35735 (local address 127.0.98.3:55825)
0928 09:01:48.357411 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.357421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.357461 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.357822 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.357823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.357824 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.357889 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.357891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.358684 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.358685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.359534 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.359537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.360179 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.360180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.360251 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.360256 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.360311 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.360348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260928 09:01:48.360850 14324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743ec7f3d70426298bf2d5d548f7120" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83440db9bf8a4040b1a2335a65af3813" is_pre_election: true
I20260928 09:01:48.360898 14458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743ec7f3d70426298bf2d5d548f7120" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f23c0d3d1664e56beca6d00861a6477" is_pre_election: true
I20260928 09:01:48.360966 14324 raft_consensus.cc:2454] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 0.
I20260928 09:01:48.361040 14458 raft_consensus.cc:2454] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 0.
I20260928 09:01:48.361179 14528 leader_election.cc:302] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7962d529ebcc42e38d4db7542a9e17d7, 83440db9bf8a4040b1a2335a65af3813; no voters:
I20260928 09:01:48.361327 14661 raft_consensus.cc:2781] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:01:48.361399 14661 raft_consensus.cc:482] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:01:48.361418 14661 raft_consensus.cc:3037] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:48.362129 14661 raft_consensus.cc:504] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.362264 14661 leader_election.cc:288] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 election: Requested vote from peers 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993), 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735)
I20260928 09:01:48.362426 14324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743ec7f3d70426298bf2d5d548f7120" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83440db9bf8a4040b1a2335a65af3813"
I20260928 09:01:48.362504 14324 raft_consensus.cc:3037] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:48.362468 14458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6743ec7f3d70426298bf2d5d548f7120" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f23c0d3d1664e56beca6d00861a6477"
I20260928 09:01:48.362572 14458 raft_consensus.cc:3037] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:48.363240 14324 raft_consensus.cc:2454] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 1.
I20260928 09:01:48.363245 14458 raft_consensus.cc:2454] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 1.
I20260928 09:01:48.363407 14528 leader_election.cc:302] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7962d529ebcc42e38d4db7542a9e17d7, 83440db9bf8a4040b1a2335a65af3813; no voters:
I20260928 09:01:48.363489 14661 raft_consensus.cc:2781] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:48.363626 14661 raft_consensus.cc:686] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7962d529ebcc42e38d4db7542a9e17d7, State: Running, Role: LEADER
I20260928 09:01:48.363713 14661 consensus_queue.cc:237] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.364511 14177 catalog_manager.cc:5495] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7962d529ebcc42e38d4db7542a9e17d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } health_report { overall_health: UNKNOWN } } }
W20260928 09:01:48.372589 14505 tablet.cc:2369] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:01:48.375780 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.373303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.0.98.3:33381)
0928 09:01:48.373420 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.373422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.373429 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:01:48.373462 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.373463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.373480 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.373548 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:01:48.373698 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.374172 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.374966 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.375191 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.375260 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.375432 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:01:48.375543 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:01:48.375642 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.375660 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.375679 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 09:01:48.377569 14458 raft_consensus.cc:1261] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Refusing update from remote peer 7962d529ebcc42e38d4db7542a9e17d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:48.377658 14324 raft_consensus.cc:1261] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Refusing update from remote peer 7962d529ebcc42e38d4db7542a9e17d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:48.377916 14665 consensus_queue.cc:1035] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:48.378032 14661 consensus_queue.cc:1035] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:48.380393 14668 mvcc.cc:205] Tried to move back new op lower bound from 7334240699912032256 to 7334240699858530304. Current Snapshot: MvccSnapshot[applied={T|T < 7334240699912032256}]
I20260928 09:01:48.381320 14670 mvcc.cc:205] Tried to move back new op lower bound from 7334240699912032256 to 7334240699858530304. Current Snapshot: MvccSnapshot[applied={T|T < 7334240699912032256}]
I20260928 09:01:48.382714 14179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:01:48.382812 14179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:01:48.383668 14179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Sending DeleteTablet for 3 replicas of tablet 6743ec7f3d70426298bf2d5d548f7120
I20260928 09:01:48.384006 14438 tablet_service.cc:1514] Processing DeleteTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54702
I20260928 09:01:48.384008 14572 tablet_service.cc:1514] Processing DeleteTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47344
I20260928 09:01:48.384022 14304 tablet_service.cc:1514] Processing DeleteTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32864
I20260928 09:01:48.384241 14677 tablet_replica.cc:330] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: stopping tablet replica
I20260928 09:01:48.384349 14677 raft_consensus.cc:2229] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:01:48.384357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14241
I20260928 09:01:48.384989 14677 raft_consensus.cc:2258] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:48.385470 14677 ts_tablet_manager.cc:1905] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:48.387452 14677 ts_tablet_manager.cc:1918] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:01:48.387531 14677 log.cc:1196] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/6743ec7f3d70426298bf2d5d548f7120
I20260928 09:01:48.387844 14677 ts_tablet_manager.cc:1939] T 6743ec7f3d70426298bf2d5d548f7120 P 6f23c0d3d1664e56beca6d00861a6477: Deleting consensus metadata
I20260928 09:01:48.388396 14166 catalog_manager.cc:4841] TS 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): tablet 6743ec7f3d70426298bf2d5d548f7120 (table test-table [id=72e0daed854a4fb18f31b198a04c9405]) successfully deleted
I20260928 09:01:48.388756 14679 tablet_replica.cc:330] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: stopping tablet replica
I20260928 09:01:48.388876 14679 raft_consensus.cc:2229] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:01:48.388981 14679 raft_consensus.cc:2258] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:48.389358 14679 ts_tablet_manager.cc:1905] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:48.390905 14679 ts_tablet_manager.cc:1918] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:01:48.391113 14679 log.cc:1196] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/6743ec7f3d70426298bf2d5d548f7120
W20260928 09:01:48.391217 14166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36993 (error 108)
I20260928 09:01:48.391465 14679 ts_tablet_manager.cc:1939] T 6743ec7f3d70426298bf2d5d548f7120 P 7962d529ebcc42e38d4db7542a9e17d7: Deleting consensus metadata
I20260928 09:01:48.391506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14375
I20260928 09:01:48.392151 14166 catalog_manager.cc:4841] TS 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): tablet 6743ec7f3d70426298bf2d5d548f7120 (table test-table [id=72e0daed854a4fb18f31b198a04c9405]) successfully deleted
I20260928 09:01:48.392593 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.392477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36993 (local address 127.0.0.1:32876)
0928 09:01:48.392520 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.392553 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 09:01:48.392740 14166 catalog_manager.cc:4584] TS 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6743ec7f3d70426298bf2d5d548f7120: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36993: connect: Connection refused (error 111)
I20260928 09:01:48.397028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14509
I20260928 09:01:48.401805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14147
I20260928 09:01:48.407181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--webserver_interface=127.0.98.62
--webserver_port=41493
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:48.490456 14681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:48.490700 14681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:48.490741 14681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:48.491879 14681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:48.491946 14681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:48.491971 14681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:48.491998 14681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:48.492027 14681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:48.493516 14681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:48.493785 14681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:48.494002 14681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:48.502727 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.502746 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.502744 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.503052 14681 server_base.cc:1034] running on GCE node
I20260928 09:01:48.503322 14681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:48.503602 14681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:48.504702 14681 hybrid_clock.cc:648] HybridClock initialized: now 1790586108504682 us; error 32 us; skew 500 ppm
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:48.517026 14681 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:48.517665 14681 webserver.cc:460] Webserver started at http://127.0.98.62:41493/ using document root <none> and password file <none>
I20260928 09:01:48.517946 14681 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:48.518057 14681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:48.519593 14681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:01:48.520325 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.520514 14681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:48.520628 14681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "7269e2e57b9f48f3be73e801cc7e2b35"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "d2311a21551c9ac777f0bf9a36084e9a"
server_key_iv: "00538fb046cbc958cb1db8f186ee34d7"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.521130 14681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:48.533021 14681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:48.535508 14681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:48.535683 14681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:48.540702 14681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42083
I20260928 09:01:48.540725 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42083 every 8 connection(s)
I20260928 09:01:48.541163 14681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:48.542629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14681
I20260928 09:01:48.542819 14750 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:01:48.542972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42083
--webserver_port=37235
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:48.543602 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap starting.
I20260928 09:01:48.551831 14750 log.cc:824] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.553933 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:01:48.554232 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap complete.
I20260928 09:01:48.556720 14750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:48.556995 14750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7269e2e57b9f48f3be73e801cc7e2b35, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.557173 14750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:48.557248 14750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:01:48.557305 14750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:01:48.557354 14750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:01:48.558377 14750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:48.558496 14750 leader_election.cc:302] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7269e2e57b9f48f3be73e801cc7e2b35; no voters:
I20260928 09:01:48.558699 14750 leader_election.cc:288] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:01:48.558737 14754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:01:48.558849 14754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 LEADER]: Becoming Leader. State: Replica: 7269e2e57b9f48f3be73e801cc7e2b35, State: Running, Role: LEADER
I20260928 09:01:48.558990 14750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:48.558997 14754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:48.559414 14756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7269e2e57b9f48f3be73e801cc7e2b35. Latest consensus state: current_term: 2 leader_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 09:01:48.559506 14756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:48.559994 14755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 09:01:48.560067 14755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:48.560969 14771 catalog_manager.cc:1259] Loaded cluster ID: 486eb4770173499dbe026e569773395b
I20260928 09:01:48.561019 14771 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: loading cluster ID for follower catalog manager: success
W20260928 09:01:48.561218 14771 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260928 09:01:48.561259 14771 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260928 09:01:48.561293 14762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:48.561667 14762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:01:48.561733 14762 catalog_manager.cc:1259] Loaded cluster ID: 486eb4770173499dbe026e569773395b
I20260928 09:01:48.561753 14762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:01:48.561895 14762 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 09:01:48.566612 14762 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 09:01:48.569450 14762 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:01:48.569667 14762 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Loaded TSK: 0
I20260928 09:01:48.569890 14762 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:01:48.630311 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:48.630540 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:48.630581 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:48.631760 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:48.631824 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:48.631877 14752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:01:48.633365 14752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37235
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:48.633616 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:48.633857 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:48.638900 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.638932 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.639050 14752 server_base.cc:1034] running on GCE node
W20260928 09:01:48.639155 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.639410 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:48.639690 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:48.640858 14752 hybrid_clock.cc:648] HybridClock initialized: now 1790586108640835 us; error 39 us; skew 500 ppm
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:48.644189 14752 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:01:48.644826 14752 webserver.cc:460] Webserver started at http://127.0.98.1:37235/ using document root <none> and password file <none>
I20260928 09:01:48.645174 14752 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:48.645260 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:48.647281 14752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:01:48.648152 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.648407 14752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:48.648506 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "83440db9bf8a4040b1a2335a65af3813"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "51fd2a4b8ba6aebbebe8de8c5fc0dbc5"
server_key_iv: "1b8dcb5471d313f3f87f6d3d36737eb5"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.649041 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:48.663358 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:48.665656 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:48.665825 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:48.666087 14752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:48.666527 14795 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:01:48.668030 14752 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:01:48.668098 14752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:01:48.668141 14752 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:01:48.668824 14752 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:01:48.668869 14752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:48.668910 14795 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Bootstrap starting.
I20260928 09:01:48.676566 14752 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36993
I20260928 09:01:48.676679 14902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36993 every 8 connection(s)
I20260928 09:01:48.677174 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:01:48.678416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14752
I20260928 09:01:48.678709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35735
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42083
--webserver_port=39699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:48.680189 14795 log.cc:824] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.682482 14795 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:01:48.682801 14795 tablet_bootstrap.cc:492] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Bootstrap complete.
I20260928 09:01:48.684043 14795 ts_tablet_manager.cc:1397] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260928 09:01:48.685547 14795 raft_consensus.cc:348] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.685941 14795 raft_consensus.cc:729] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83440db9bf8a4040b1a2335a65af3813, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.686134 14795 consensus_queue.cc:260] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } }
I20260928 09:01:48.686462 14795 ts_tablet_manager.cc:1428] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:48.693585 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.677989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:42027)
0928 09:01:48.680661 (+ 2672us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.682633 (+ 1972us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.683457 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.684050 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.684054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.684396 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:48.684707 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.684714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.686369 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.686370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.687253 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.687255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.687929 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.687929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.688054 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.688609 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:48.688618 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:48.690012 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:48.691900 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.691907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.691909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.692128 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.692382 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.692382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.692383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.692491 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.692964 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:48.692971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:48.693105 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.693199 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.693383 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2598,"thread_start_us":95,"threads_started":1}
I20260928 09:01:48.693593 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.678076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42027 (local address 127.0.98.62:42083)
0928 09:01:48.680131 (+ 2055us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.680141 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.682786 (+ 2645us) server_negotiation.cc:407] Connection header received
0928 09:01:48.683687 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.683695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.683755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.683869 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.684860 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.686247 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.687364 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.687769 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.687963 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.690149 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.690153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.690153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.690163 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.691801 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.692214 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.692222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.692222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.692310 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.692565 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.692566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.692566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.692893 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.692987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.693266 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.693387 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":116,"threads_started":1}
I20260928 09:01:48.695050 14903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 09:01:48.695145 14903 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:48.695335 14903 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 09:01:48.696007 14713 ts_manager.cc:194] Registered new tserver with Master: 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993)
I20260928 09:01:48.697111 14713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42027
I20260928 09:01:48.698114 14903 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 09:01:48.705483 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.696569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36993 (local address 127.0.0.1:32882)
0928 09:01:48.697425 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.697461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.697518 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.697796 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.697802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.698128 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:48.698186 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.698189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.699392 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.699393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.700008 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.700010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.700481 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.700484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.700561 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.701114 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:48.701124 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:48.702420 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:48.704218 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.704225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.704227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.704429 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.704669 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.704670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.704670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.704737 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.705057 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:48.705064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:48.705119 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.705229 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.705369 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":148,"threads_started":1}
I20260928 09:01:48.705920 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.696867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.0.98.1:36993)
0928 09:01:48.697639 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.697643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.697652 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:01:48.697683 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.697684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.697698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.697756 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.698304 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.699300 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.700108 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.700397 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.700482 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.702524 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.702532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.702532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.702540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.704113 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.704519 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.704522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.704523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.704582 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.704835 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.704836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.704836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.704981 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.705065 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.705736 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.705833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":68,"threads_started":1}
I20260928 09:01:48.706269 14837 tablet_service.cc:1514] Processing DeleteTablet for tablet 6743ec7f3d70426298bf2d5d548f7120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32882
I20260928 09:01:48.706818 14912 tablet_replica.cc:330] stopping tablet replica
I20260928 09:01:48.706928 14912 raft_consensus.cc:2229] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:01:48.706996 14912 raft_consensus.cc:2258] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:48.707307 14912 ts_tablet_manager.cc:1905] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:48.709337 14912 ts_tablet_manager.cc:1918] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:01:48.709429 14912 log.cc:1196] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/6743ec7f3d70426298bf2d5d548f7120
I20260928 09:01:48.709650 14912 ts_tablet_manager.cc:1939] T 6743ec7f3d70426298bf2d5d548f7120 P 83440db9bf8a4040b1a2335a65af3813: Deleting consensus metadata
I20260928 09:01:48.710177 14701 catalog_manager.cc:4841] TS 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993): tablet 6743ec7f3d70426298bf2d5d548f7120 (table test-table [id=72e0daed854a4fb18f31b198a04c9405]) successfully deleted
W20260928 09:01:48.769647 14907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:48.769860 14907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:48.769883 14907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:48.771075 14907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:48.771118 14907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:48.771144 14907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:01:48.772590 14907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39699
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:48.772850 14907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:48.773083 14907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:48.778164 14917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.778173 14918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.778170 14920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.778555 14907 server_base.cc:1034] running on GCE node
I20260928 09:01:48.778779 14907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:48.778997 14907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:48.780164 14907 hybrid_clock.cc:648] HybridClock initialized: now 1790586108780156 us; error 39 us; skew 500 ppm
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:48.782953 14907 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:01:48.783413 14907 webserver.cc:460] Webserver started at http://127.0.98.2:39699/ using document root <none> and password file <none>
I20260928 09:01:48.783639 14907 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:48.783717 14907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:48.785197 14907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:01:48.785781 14927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.785946 14907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:01:48.786044 14907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "6f23c0d3d1664e56beca6d00861a6477"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "90119dc480417d9dc4e6cbbafab8debe"
server_key_iv: "c06a219844c036870fb3a69ad6795468"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.786396 14907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:48.815409 14907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:48.817868 14907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:48.818037 14907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:48.818287 14907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:48.818598 14907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:48.818647 14907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.818688 14907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:48.818728 14907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.824612 14907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35735
I20260928 09:01:48.824707 15040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35735 every 8 connection(s)
I20260928 09:01:48.825116 14907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:48.834764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 14907
I20260928 09:01:48.835067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33381
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42083
--webserver_port=36721
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:48.837129 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.825845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59611 (local address 127.0.98.62:42083)
0928 09:01:48.825907 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.825909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.826140 (+ 231us) server_negotiation.cc:407] Connection header received
0928 09:01:48.826628 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.826629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.826639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.826691 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.828365 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.829077 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.830707 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.830920 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.830990 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.833773 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.833774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.833775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.833782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.835449 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.835903 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.835905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.835906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.835968 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.836310 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.836312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.836312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.836478 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.836529 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.836987 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.837063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:01:48.837442 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.825877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:59611)
0928 09:01:48.826014 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.826050 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.826536 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.826746 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.826750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.827675 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:48.828125 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.828136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.829179 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.829180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.830594 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.830597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.831022 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.831024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.831205 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.831829 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:48.831840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:48.833668 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:48.835553 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.835560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.835563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.835806 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.836038 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.836039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.836040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.836192 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.836584 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:48.836591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:48.836814 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.836941 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.837136 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260928 09:01:48.838102 15041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 09:01:48.838197 15041 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:48.838383 15041 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 09:01:48.838806 14713 ts_manager.cc:194] Registered new tserver with Master: 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735)
I20260928 09:01:48.839455 14713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59611
W20260928 09:01:48.917631 15044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:48.917855 15044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:48.917879 15044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:48.918991 15044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:48.919025 15044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:48.919051 15044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:01:48.920403 15044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36721
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:48.920585 15044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:48.920837 15044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:48.925918 15050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.925967 15052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:48.925937 15049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:48.926189 15044 server_base.cc:1034] running on GCE node
I20260928 09:01:48.926403 15044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:48.926649 15044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:48.927817 15044 hybrid_clock.cc:648] HybridClock initialized: now 1790586108927808 us; error 33 us; skew 500 ppm
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:48.930437 15044 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:01:48.930817 15044 webserver.cc:460] Webserver started at http://127.0.98.3:36721/ using document root <none> and password file <none>
I20260928 09:01:48.931011 15044 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:48.931063 15044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:48.932338 15044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:48.933010 15059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.933199 15044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:01:48.933328 15044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "7962d529ebcc42e38d4db7542a9e17d7"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "1f7e97729d0daaea14be8ae9946a7654"
server_key_iv: "0402990bb9da6a6d722c9db95d9c44d5"
server_key_version: "encryptionkey@0"
I20260928 09:01:48.933668 15044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:48.947222 15044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:48.949237 15044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:48.949348 15044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:48.949527 15044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:48.949790 15044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:48.949819 15044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.949843 15044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:48.949855 15044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:48.955818 15044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33381
I20260928 09:01:48.955875 15172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33381 every 8 connection(s)
I20260928 09:01:48.956247 15044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:01:48.960773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15044
Sep 28 09:01:48 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:01:48.966611 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.957018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35925 (local address 127.0.98.62:42083)
0928 09:01:48.957067 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.957069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.957342 (+ 273us) server_negotiation.cc:407] Connection header received
0928 09:01:48.957872 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.957873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.957886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.957939 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.958974 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.959587 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.960816 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.961102 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.961169 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.963501 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.963502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.963503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.963511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.965036 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.965494 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.965495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.965495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.965547 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.965811 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.965812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.965812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.965926 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.965988 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.966384 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.966496 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:01:48.966768 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.956991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:35925)
0928 09:01:48.957171 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.957255 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.957759 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.958007 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.958017 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.958655 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:48.958860 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.958866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.959680 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.959681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.960719 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.960725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.961280 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.961281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.961387 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.961935 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:48.961944 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:48.963392 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:48.965108 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.965115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.965117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.965421 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.965613 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:48.965614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:48.965615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:48.965728 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:48.965998 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:48.966005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:48.966172 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.966300 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.966482 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260928 09:01:48.967396 15173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42083
I20260928 09:01:48.967517 15173 heartbeater.cc:464] Registering TS with master...
I20260928 09:01:48.967723 15173 heartbeater.cc:511] Master 127.0.98.62:42083 requested a full tablet report, sending...
I20260928 09:01:48.968182 14713 ts_manager.cc:194] Registered new tserver with Master: 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
I20260928 09:01:48.968416 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.961169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.0.98.62:42083)
0928 09:01:48.961270 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.961272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.961314 (+ 42us) server_negotiation.cc:407] Connection header received
0928 09:01:48.961390 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.961391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.961404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.961462 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:48.962065 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.962635 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.963467 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.963679 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.963746 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.964845 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:48.964846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:48.964846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:48.964853 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:48.966421 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.966866 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.966867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.966868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.966926 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:48.967124 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:48.967125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:48.967125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:48.967279 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:48.967499 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.968081 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.968163 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
I20260928 09:01:48.968783 14713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35925
I20260928 09:01:48.969204 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:01:48.971633 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.969561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.0.98.62:42083)
0928 09:01:48.969600 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.969602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.969705 (+ 103us) server_negotiation.cc:407] Connection header received
0928 09:01:48.969760 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.969761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.969788 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.969851 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:01:48.969928 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.970371 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.971050 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.971225 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.971271 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.971390 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:01:48.971476 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:01:48.971542 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.971571 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.971590 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:01:48.971901 14713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:01:48.973634 14713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:01:48.975845 14837 tablet_service.cc:1467] Processing CreateTablet for tablet 945876ba534b43199d9bbe8cffec2e8c (DEFAULT_TABLE table=test-table [id=2aec2e9ff68b4b6088a5dbabeed2835c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:48.976064 14837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945876ba534b43199d9bbe8cffec2e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:48.977679 14795 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Bootstrap starting.
I20260928 09:01:48.977982 14795 tablet_bootstrap.cc:654] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:48.978528 14795 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: No bootstrap required, opened a new log
I20260928 09:01:48.978588 14795 ts_tablet_manager.cc:1397] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:48.978919 14795 raft_consensus.cc:348] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.979000 14795 raft_consensus.cc:374] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:48.979051 14795 raft_consensus.cc:729] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83440db9bf8a4040b1a2335a65af3813, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.979095 14795 consensus_queue.cc:260] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.979195 14795 ts_tablet_manager.cc:1428] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:48.979341 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.975309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35735 (local address 127.0.0.1:54712)
0928 09:01:48.975375 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.975389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.975437 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.975773 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.975774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.975776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.975818 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.975820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.977012 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.977013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.977927 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.977929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.979102 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.979103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.979173 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.979196 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.979227 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.979272 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 09:01:48.979801 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.975409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33381 (local address 127.0.0.1:47352)
0928 09:01:48.975874 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.975885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.975979 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.976243 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.976244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.976246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.976304 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.976305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.977529 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.977530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.978624 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.978627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.979578 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.979579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.979646 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.979649 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.979695 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.979736 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":108,"threads_started":1}
I20260928 09:01:48.979979 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.975431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.0.98.2:35735)
0928 09:01:48.975587 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.975592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.975601 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:01:48.975640 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.975642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.975654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.975752 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.975912 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.976923 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.978255 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.979011 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.979656 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.979719 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.979745 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.979783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 09:01:48.980103 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.975493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.0.98.3:33381)
0928 09:01:48.975743 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.975748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.975934 (+ 186us) server_negotiation.cc:407] Connection header received
0928 09:01:48.976079 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.976082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.976098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.976166 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.976389 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.977424 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.978714 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.979421 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.979486 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.979545 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.979968 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.980018 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260928 09:01:48.980733 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 945876ba534b43199d9bbe8cffec2e8c (DEFAULT_TABLE table=test-table [id=2aec2e9ff68b4b6088a5dbabeed2835c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:48.980739 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 945876ba534b43199d9bbe8cffec2e8c (DEFAULT_TABLE table=test-table [id=2aec2e9ff68b4b6088a5dbabeed2835c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:01:48.981081 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945876ba534b43199d9bbe8cffec2e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:48.981086 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945876ba534b43199d9bbe8cffec2e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:01:48.983815 15184 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Bootstrap starting.
I20260928 09:01:48.983924 15183 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Bootstrap starting.
I20260928 09:01:48.984457 15184 tablet_bootstrap.cc:654] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:48.984514 15183 tablet_bootstrap.cc:654] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Neither blocks nor log segments found. Creating new log.
I20260928 09:01:48.984997 15184 log.cc:824] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.985059 15183 log.cc:824] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:48.985702 15184 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: No bootstrap required, opened a new log
I20260928 09:01:48.985714 15183 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: No bootstrap required, opened a new log
I20260928 09:01:48.985756 15184 ts_tablet_manager.cc:1397] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:48.985790 15183 ts_tablet_manager.cc:1397] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:01:48.987303 15183 raft_consensus.cc:348] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.987309 15184 raft_consensus.cc:348] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.987447 15183 raft_consensus.cc:374] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:48.987464 15184 raft_consensus.cc:374] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:01:48.987471 15183 raft_consensus.cc:729] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7962d529ebcc42e38d4db7542a9e17d7, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.987514 15184 raft_consensus.cc:729] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f23c0d3d1664e56beca6d00861a6477, State: Initialized, Role: FOLLOWER
I20260928 09:01:48.987622 15184 consensus_queue.cc:260] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.987620 15183 consensus_queue.cc:260] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.987915 15183 ts_tablet_manager.cc:1428] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:01:48.987915 15184 ts_tablet_manager.cc:1428] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:01:48.987922 15041 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 09:01:48.988044 15173 heartbeater.cc:503] Master 127.0.98.62:42083 was elected leader, sending a full tablet report...
I20260928 09:01:48.994872 15188 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:48.994982 15188 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:48.995249 15188 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993)
I20260928 09:01:48.998922 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.995542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36019 (local address 127.0.98.2:35735)
0928 09:01:48.995607 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.995609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.995620 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:01:48.995645 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.995646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.995657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.995712 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.995891 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.996622 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.998004 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.998568 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.998647 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.998659 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.998788 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.998829 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:01:48.998921 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.995449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35735 (local address 127.0.98.3:36019)
0928 09:01:48.995511 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.995526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.995579 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.995731 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.995733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.995735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.995808 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.995810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.996719 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.996720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.997911 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.997914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.998644 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.998644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.998702 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.998705 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.998778 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.998822 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":5}
I20260928 09:01:48.999042 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.995640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59791 (local address 127.0.98.1:36993)
0928 09:01:48.995691 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:48.995693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:48.995767 (+ 74us) server_negotiation.cc:407] Connection header received
0928 09:01:48.995845 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:48.995846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:48.995883 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:48.995949 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:01:48.996067 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.996817 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.998070 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.998705 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.998776 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.998859 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:48.998919 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 09:01:48.998969 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:01:48.999224 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:48.995565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36993 (local address 127.0.98.3:59791)
0928 09:01:48.995711 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:48.995722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:48.995780 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:48.995955 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:48.995955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:48.995956 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:01:48.996004 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.996005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.996890 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.996891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.997995 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:48.997997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:48.998776 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:48.998776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:48.998835 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:48.998838 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:01:48.998911 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 09:01:48.998950 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260928 09:01:48.999286 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945876ba534b43199d9bbe8cffec2e8c" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f23c0d3d1664e56beca6d00861a6477" is_pre_election: true
I20260928 09:01:48.999500 14995 raft_consensus.cc:2454] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 0.
I20260928 09:01:48.999509 14857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945876ba534b43199d9bbe8cffec2e8c" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83440db9bf8a4040b1a2335a65af3813" is_pre_election: true
I20260928 09:01:48.999637 14857 raft_consensus.cc:2454] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 0.
I20260928 09:01:48.999722 15063 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f23c0d3d1664e56beca6d00861a6477, 7962d529ebcc42e38d4db7542a9e17d7; no voters:
I20260928 09:01:48.999861 15188 raft_consensus.cc:2781] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:01:48.999930 15188 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:01:48.999949 15188 raft_consensus.cc:3037] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:49.000696 15188 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:49.000831 15188 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 election: Requested vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993)
I20260928 09:01:49.000998 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945876ba534b43199d9bbe8cffec2e8c" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f23c0d3d1664e56beca6d00861a6477"
I20260928 09:01:49.001036 14857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945876ba534b43199d9bbe8cffec2e8c" candidate_uuid: "7962d529ebcc42e38d4db7542a9e17d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83440db9bf8a4040b1a2335a65af3813"
I20260928 09:01:49.001094 14995 raft_consensus.cc:3037] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:49.001101 14857 raft_consensus.cc:3037] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:01:49.001788 14857 raft_consensus.cc:2454] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 1.
I20260928 09:01:49.001878 14995 raft_consensus.cc:2454] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7962d529ebcc42e38d4db7542a9e17d7 in term 1.
I20260928 09:01:49.001950 15063 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7962d529ebcc42e38d4db7542a9e17d7, 83440db9bf8a4040b1a2335a65af3813; no voters:
I20260928 09:01:49.002068 15188 raft_consensus.cc:2781] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:01:49.002245 15188 raft_consensus.cc:686] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7962d529ebcc42e38d4db7542a9e17d7, State: Running, Role: LEADER
I20260928 09:01:49.002343 15188 consensus_queue.cc:237] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:49.003156 14714 catalog_manager.cc:5495] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7962d529ebcc42e38d4db7542a9e17d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260928 09:01:49.015084 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.012714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.0.98.3:33381)
0928 09:01:49.012775 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:49.012778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:49.012789 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:01:49.012835 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:49.012836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:49.012875 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:49.012956 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:01:49.013062 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:49.013537 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:49.014272 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:49.014480 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:49.014563 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:49.014723 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:01:49.014838 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:01:49.014931 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:49.014964 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 09:01:49.014987 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:01:49.017104 14857 raft_consensus.cc:1261] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Refusing update from remote peer 7962d529ebcc42e38d4db7542a9e17d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:49.017113 14995 raft_consensus.cc:1261] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Refusing update from remote peer 7962d529ebcc42e38d4db7542a9e17d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:01:49.017463 15190 consensus_queue.cc:1035] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:49.017596 15188 consensus_queue.cc:1035] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:01:49.019682 15192 mvcc.cc:205] Tried to move back new op lower bound from 7334240702530912256 to 7334240702474436608. Current Snapshot: MvccSnapshot[applied={T|T < 7334240702530912256}]
I20260928 09:01:49.020182 15193 mvcc.cc:205] Tried to move back new op lower bound from 7334240702530912256 to 7334240702474436608. Current Snapshot: MvccSnapshot[applied={T|T < 7334240702530912256 or (T in {7334240702530912256})}]
I20260928 09:01:49.020457 15194 mvcc.cc:205] Tried to move back new op lower bound from 7334240702530912256 to 7334240702474436608. Current Snapshot: MvccSnapshot[applied={T|T < 7334240702530912256}]
I20260928 09:01:49.022657 14713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:01:49.022766 14713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:01:49.023494 14713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Sending DeleteTablet for 3 replicas of tablet 945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:49.023836 14975 tablet_service.cc:1514] Processing DeleteTablet for tablet 945876ba534b43199d9bbe8cffec2e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54712
I20260928 09:01:49.023869 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet 945876ba534b43199d9bbe8cffec2e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47352
I20260928 09:01:49.024049 14837 tablet_service.cc:1514] Processing DeleteTablet for tablet 945876ba534b43199d9bbe8cffec2e8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:01:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32882
I20260928 09:01:49.024092 15202 tablet_replica.cc:330] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: stopping tablet replica
I20260928 09:01:49.024154 14912 tablet_replica.cc:330] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: stopping tablet replica
I20260928 09:01:49.024178 15202 raft_consensus.cc:2229] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:01:49.024290 14912 raft_consensus.cc:2229] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:01:49.024341 15202 raft_consensus.cc:2258] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:49.024381 14912 raft_consensus.cc:2258] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:49.024704 15201 tablet_replica.cc:330] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: stopping tablet replica
I20260928 09:01:49.024716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14752
I20260928 09:01:49.024770 14912 ts_tablet_manager.cc:1905] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:49.025439 15201 raft_consensus.cc:2229] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:01:49.025580 15201 raft_consensus.cc:2258] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:01:49.025859 15202 ts_tablet_manager.cc:1905] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:49.026904 15201 ts_tablet_manager.cc:1905] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:01:49.027463 15202 ts_tablet_manager.cc:1918] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:01:49.027542 15202 log.cc:1196] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:49.027848 15202 ts_tablet_manager.cc:1939] T 945876ba534b43199d9bbe8cffec2e8c P 7962d529ebcc42e38d4db7542a9e17d7: Deleting consensus metadata
I20260928 09:01:49.028285 14701 catalog_manager.cc:4841] TS 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): tablet 945876ba534b43199d9bbe8cffec2e8c (table test-table [id=2aec2e9ff68b4b6088a5dbabeed2835c]) successfully deleted
I20260928 09:01:49.029067 15201 ts_tablet_manager.cc:1918] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:01:49.029155 15201 log.cc:1196] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:49.029435 15201 ts_tablet_manager.cc:1939] T 945876ba534b43199d9bbe8cffec2e8c P 6f23c0d3d1664e56beca6d00861a6477: Deleting consensus metadata
I20260928 09:01:49.029820 14701 catalog_manager.cc:4841] TS 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): tablet 945876ba534b43199d9bbe8cffec2e8c (table test-table [id=2aec2e9ff68b4b6088a5dbabeed2835c]) successfully deleted
W20260928 09:01:49.032406 14701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36993 (error 108)
I20260928 09:01:49.032789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14907
I20260928 09:01:49.032960 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.032818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36993 (local address 127.0.0.1:32884)
0928 09:01:49.032880 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.032920 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 09:01:49.033502 14701 catalog_manager.cc:4584] TS 83440db9bf8a4040b1a2335a65af3813 (127.0.98.1:36993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 945876ba534b43199d9bbe8cffec2e8c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36993: connect: Connection refused (error 111)
I20260928 09:01:49.038913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15044
I20260928 09:01:49.043071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 14681
I20260928 09:01:49.048449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--webserver_interface=127.0.98.62
--webserver_port=41493
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:49.129166 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:49.129381 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:49.129405 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:49.130579 15204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:01:49.130620 15204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:49.130635 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:49.130647 15204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:01:49.130659 15204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:01:49.132141 15204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:49.132396 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:49.132681 15204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:49.137491 15210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.137533 15212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.137501 15209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:49.137909 15204 server_base.cc:1034] running on GCE node
I20260928 09:01:49.138145 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:49.138449 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:49.139608 15204 hybrid_clock.cc:648] HybridClock initialized: now 1790586109139602 us; error 34 us; skew 500 ppm
Sep 28 09:01:49 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:49.142118 15204 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:01:49.142584 15204 webserver.cc:460] Webserver started at http://127.0.98.62:41493/ using document root <none> and password file <none>
I20260928 09:01:49.142802 15204 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:49.142845 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:49.144189 15204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:49.144856 15219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.145042 15204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:49.145131 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "7269e2e57b9f48f3be73e801cc7e2b35"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "d2311a21551c9ac777f0bf9a36084e9a"
server_key_iv: "00538fb046cbc958cb1db8f186ee34d7"
server_key_version: "encryptionkey@0"
I20260928 09:01:49.145499 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:49.168581 15204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:49.170949 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:49.171101 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:49.175328 15204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42083
I20260928 09:01:49.175356 15271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42083 every 8 connection(s)
I20260928 09:01:49.175776 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:01:49.177431 15272 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:01:49.178010 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap starting.
I20260928 09:01:49.183517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15204
I20260928 09:01:49.183785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42083
--webserver_port=37235
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:01:49.186214 15272 log.cc:824] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:49.188450 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:01:49.188828 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35: Bootstrap complete.
I20260928 09:01:49.191110 15272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:49.191354 15272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7269e2e57b9f48f3be73e801cc7e2b35, State: Initialized, Role: FOLLOWER
I20260928 09:01:49.191540 15272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:49.191613 15272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:01:49.191664 15272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:01:49.191710 15272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 2 FOLLOWER]: Advancing to term 3
I20260928 09:01:49.192692 15272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:49.192808 15272 leader_election.cc:302] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7269e2e57b9f48f3be73e801cc7e2b35; no voters:
I20260928 09:01:49.192986 15272 leader_election.cc:288] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 09:01:49.193012 15276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 09:01:49.193118 15276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [term 3 LEADER]: Becoming Leader. State: Replica: 7269e2e57b9f48f3be73e801cc7e2b35, State: Running, Role: LEADER
I20260928 09:01:49.193200 15276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } }
I20260928 09:01:49.193245 15272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:01:49.193427 15276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7269e2e57b9f48f3be73e801cc7e2b35. Latest consensus state: current_term: 3 leader_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 09:01:49.193490 15276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:49.193651 15280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:01:49.193619 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7269e2e57b9f48f3be73e801cc7e2b35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42083 } } }
I20260928 09:01:49.193876 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7269e2e57b9f48f3be73e801cc7e2b35 [sys.catalog]: This master's current role is: LEADER
I20260928 09:01:49.194494 15280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:01:49.194609 15280 catalog_manager.cc:1259] Loaded cluster ID: 486eb4770173499dbe026e569773395b
I20260928 09:01:49.194653 15280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 09:01:49.195040 15280 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdd3ac529ad google::LogMessage::Fail() at ??:0
@ 0x7fdd3ac5588a google::LogMessage::SendToLog() at ??:0
@ 0x7fdd3ac524de google::LogMessage::Flush() at ??:0
@ 0x7fdd3ac53a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdd42b730f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdd42b7c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdd3cb200d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdd3cb15954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdd3d9796db start_thread at ??:0
@ 0x7fdd3975061f clone at ??:0
W20260928 09:01:49.280745 15274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:49.280952 15274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:49.280977 15274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:49.282173 15274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:49.282217 15274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:49.282245 15274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:01:49.283701 15274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37235
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:49.283934 15274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:49.284178 15274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:49.289314 15309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.289399 15308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.289328 15311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:49.289987 15274 server_base.cc:1034] running on GCE node
I20260928 09:01:49.290217 15274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:49.290453 15274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:49.291610 15274 hybrid_clock.cc:648] HybridClock initialized: now 1790586109291597 us; error 30 us; skew 500 ppm
Sep 28 09:01:49 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:49.294170 15274 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:01:49.294605 15274 webserver.cc:460] Webserver started at http://127.0.98.1:37235/ using document root <none> and password file <none>
I20260928 09:01:49.294827 15274 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:49.294896 15274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:49.296276 15274 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:49.297099 15318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.297277 15274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:01:49.297402 15274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "83440db9bf8a4040b1a2335a65af3813"
format_stamp: "Formatted at 2026-09-28 09:01:47 on dist-test-slave-286n"
server_key: "51fd2a4b8ba6aebbebe8de8c5fc0dbc5"
server_key_iv: "1b8dcb5471d313f3f87f6d3d36737eb5"
server_key_version: "encryptionkey@0"
I20260928 09:01:49.297746 15274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:49.370697 15274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:49.373081 15274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:49.373243 15274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:49.373492 15274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:49.373979 15325 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:01:49.375962 15274 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:01:49.376032 15274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:01:49.376077 15274 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:01:49.376819 15274 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:01:49.376870 15274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:49.376952 15325 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Bootstrap starting.
I20260928 09:01:49.378847 15325 log.cc:824] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Log is configured to *not* fsync() on all Append() calls
I20260928 09:01:49.381832 15325 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:01:49.382153 15325 tablet_bootstrap.cc:492] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Bootstrap complete.
I20260928 09:01:49.382313 15325 ts_tablet_manager.cc:1397] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 09:01:49.383713 15274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36993
I20260928 09:01:49.384083 15274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:01:49.384148 15325 raft_consensus.cc:348] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:49.384620 15325 raft_consensus.cc:729] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83440db9bf8a4040b1a2335a65af3813, State: Initialized, Role: FOLLOWER
I20260928 09:01:49.384552 15433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36993 every 8 connection(s)
I20260928 09:01:49.384821 15325 consensus_queue.cc:260] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:49.385114 15325 ts_tablet_manager.cc:1428] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 09:01:49.389958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15274
I20260928 09:01:49.390211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35735
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42083
--webserver_port=39699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:49.420331 15436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:01:49.384843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:57549)
0928 09:01:49.385189 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.385231 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:49.385368 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:49.420057 (+ 34689us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":108,"threads_started":1}
W20260928 09:01:49.420650 15434 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42083: Client connection negotiation failed: client connection to 127.0.98.62:42083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:01:49.421089 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.420926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:56615)
0928 09:01:49.420995 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.421059 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 09:01:49.421459 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.421359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:43013)
0928 09:01:49.421393 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.421419 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 09:01:49.421579 15434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:01:49.475499 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:49.475713 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:49.475737 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:49.476887 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:49.476924 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:49.476950 15438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:01:49.478395 15438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39699
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:49.478600 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:49.478788 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:01:49.483906 15438 server_base.cc:1034] running on GCE node
W20260928 09:01:49.483850 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.483839 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.483839 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:49.484258 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:49.484503 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:49.485683 15438 hybrid_clock.cc:648] HybridClock initialized: now 1790586109485634 us; error 39 us; skew 500 ppm
Sep 28 09:01:49 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:49.488109 15438 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:01:49.488555 15438 webserver.cc:460] Webserver started at http://127.0.98.2:39699/ using document root <none> and password file <none>
I20260928 09:01:49.488812 15438 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:49.488866 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:49.490152 15438 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:49.490765 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.490971 15438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:01:49.491056 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "6f23c0d3d1664e56beca6d00861a6477"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "90119dc480417d9dc4e6cbbafab8debe"
server_key_iv: "c06a219844c036870fb3a69ad6795468"
server_key_version: "encryptionkey@0"
I20260928 09:01:49.491400 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:49.531410 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:49.533550 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:49.533684 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:49.533938 15438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:49.534243 15438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:49.534294 15438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.534336 15438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:49.534363 15438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.540156 15438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35735
I20260928 09:01:49.540210 15566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35735 every 8 connection(s)
I20260928 09:01:49.540583 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:01:49.541652 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.541283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:39909)
0928 09:01:49.541440 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.541511 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260928 09:01:49.542060 15567 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42083: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
I20260928 09:01:49.542337 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.542245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:53729)
0928 09:01:49.542291 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.542318 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 09:01:49.542613 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.542542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:36857)
0928 09:01:49.542577 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.542598 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 09:01:49.542713 15567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:01:49.545859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15438
I20260928 09:01:49.546056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33381
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42083
--webserver_port=36721
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:01:49.625692 15570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:01:49.625942 15570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:01:49.625983 15570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:01:49.627146 15570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:01:49.627215 15570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:01:49.627334 15570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:01:49.629047 15570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36721
--tserver_master_addrs=127.0.98.62:42083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:01:49.629315 15570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:01:49.629566 15570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:01:49.634696 15578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:49.634743 15570 server_base.cc:1034] running on GCE node
W20260928 09:01:49.634711 15576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:01:49.634778 15575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:01:49.635175 15570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:01:49.635437 15570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:01:49.636580 15570 hybrid_clock.cc:648] HybridClock initialized: now 1790586109636567 us; error 28 us; skew 500 ppm
Sep 28 09:01:49 dist-test-slave-286n krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:01:49.639113 15570 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:01:49.639592 15570 webserver.cc:460] Webserver started at http://127.0.98.3:36721/ using document root <none> and password file <none>
I20260928 09:01:49.639830 15570 fs_manager.cc:359] Metadata directory not provided
I20260928 09:01:49.639901 15570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:01:49.641335 15570 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:01:49.641891 15585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.642064 15570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:01:49.642162 15570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "7962d529ebcc42e38d4db7542a9e17d7"
format_stamp: "Formatted at 2026-09-28 09:01:48 on dist-test-slave-286n"
server_key: "1f7e97729d0daaea14be8ae9946a7654"
server_key_iv: "0402990bb9da6a6d722c9db95d9c44d5"
server_key_version: "encryptionkey@0"
I20260928 09:01:49.642516 15570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:01:49.670014 15570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:01:49.672477 15570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:01:49.672662 15570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:01:49.672926 15570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:01:49.673252 15570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:01:49.673308 15570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.673350 15570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:01:49.673390 15570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:01:49.679090 15570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33381
I20260928 09:01:49.679134 15698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33381 every 8 connection(s)
I20260928 09:01:49.679510 15570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:01:49.680629 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.680260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:33199)
0928 09:01:49.680407 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.680476 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 09:01:49.680967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15570
W20260928 09:01:49.681164 15699 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42083: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
I20260928 09:01:49.681456 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.681339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:55013)
0928 09:01:49.681405 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.681432 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 09:01:49.681763 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:49.681708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:39629)
0928 09:01:49.681730 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:49.681749 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 09:01:49.681880 15699 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:01:50.422166 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:50.421936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:60123)
0928 09:01:50.422061 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:50.422100 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260928 09:01:50.543269 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:50.543057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:45733)
0928 09:01:50.543174 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:50.543216 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 09:01:50.682395 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:50.682201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:33033)
0928 09:01:50.682310 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:50.682347 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260928 09:01:51.229543 15706 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:51.229682 15706 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:51.229974 15706 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
Sep 28 09:01:51 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 09:01:51 dist-test-slave-286n krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 09:01:51.240705 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.230139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35735 (local address 127.0.98.1:50809)
0928 09:01:51.230245 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:51.230256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:51.230350 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:51.231161 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:51.231165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:51.231546 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:51.231949 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:51.231952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.233244 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.233245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:51.233875 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:51.233877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.234468 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.234469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:51.234609 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:51.235251 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:51.235256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:51.237131 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:51.239155 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:51.239157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:51.239376 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:51.239609 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:51.239610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:51.239666 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:51.240095 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:51.240103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:51.240272 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 09:01:51.240375 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 09:01:51.240556 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":8,"thread_start_us":54,"threads_started":1}
I20260928 09:01:51.240710 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.230420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34405 (local address 127.0.98.3:33381)
0928 09:01:51.230542 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:51.230549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:51.230565 (+ 16us) server_negotiation.cc:407] Connection header received
0928 09:01:51.230998 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:51.231000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:51.231036 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:51.231213 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:51.231897 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.232885 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.233973 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.234373 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.234495 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:51.236874 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:51.236879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:51.236879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:51.236896 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:51.238855 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239317 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:51.239319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:51.239320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:51.239427 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239691 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:51.239692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:51.239692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:51.240019 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:51.240108 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:51.240444 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 09:01:51.240580 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260928 09:01:51.240731 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.230257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50809 (local address 127.0.98.2:35735)
0928 09:01:51.230422 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 09:01:51.230434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:01:51.230459 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:01:51.230937 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:01:51.230939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:01:51.230982 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:01:51.231093 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:01:51.232051 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.233116 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.233955 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.234372 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.234501 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:51.237228 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:01:51.237232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:01:51.237232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:01:51.237249 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:01:51.239085 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239446 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:51.239448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:51.239448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:51.239544 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239741 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:01:51.239742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:01:51.239743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:01:51.240019 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:01:51.240108 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:01:51.240451 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 09:01:51.240580 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260928 09:01:51.241015 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.230305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33381 (local address 127.0.98.1:34405)
0928 09:01:51.230398 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:51.230409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:01:51.230459 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:01:51.231205 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:01:51.231206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:01:51.231544 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:01:51.231763 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:51.231769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.232973 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.232974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:51.233877 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:01:51.233880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:01:51.234471 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:01:51.234472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:01:51.234597 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:01:51.235203 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:01:51.235214 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:01:51.236762 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:01:51.238945 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:51.238951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:51.238953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:51.239220 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:51.239499 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:01:51.239499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:01:51.239500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:01:51.239623 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:01:51.240095 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:01:51.240103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:01:51.240257 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 09:01:51.240375 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 09:01:51.240608 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":142,"thread_start_us":46,"threads_started":1}
W20260928 09:01:51.241788 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
W20260928 09:01:51.241864 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:51.241916 15322 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83440db9bf8a4040b1a2335a65af3813; no voters: 6f23c0d3d1664e56beca6d00861a6477, 7962d529ebcc42e38d4db7542a9e17d7
I20260928 09:01:51.242039 15706 raft_consensus.cc:2726] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:01:51.422793 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.422630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:46773)
0928 09:01:51.422706 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:51.422740 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 09:01:51.543967 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.543716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:48391)
0928 09:01:51.543862 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:51.543912 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 09:01:51.683074 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:51.682863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:55871)
0928 09:01:51.682985 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:51.683029 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260928 09:01:52.423518 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:52.423272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:44863)
0928 09:01:52.423429 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:52.423467 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260928 09:01:52.544775 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:52.544460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:53623)
0928 09:01:52.544619 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:52.544727 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260928 09:01:52.683732 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:52.683527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:46953)
0928 09:01:52.683653 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:52.683689 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260928 09:01:52.898005 15716 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:52.898123 15716 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:52.898308 15716 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
W20260928 09:01:52.898882 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
W20260928 09:01:52.898955 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:52.898985 15322 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83440db9bf8a4040b1a2335a65af3813; no voters: 6f23c0d3d1664e56beca6d00861a6477, 7962d529ebcc42e38d4db7542a9e17d7
I20260928 09:01:52.899072 15716 raft_consensus.cc:2726] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:01:53.424279 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:53.424082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:53279)
0928 09:01:53.424209 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:53.424234 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 09:01:53.545543 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:53.545298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:58309)
0928 09:01:53.545436 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:53.545481 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260928 09:01:53.684437 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:53.684190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:38481)
0928 09:01:53.684305 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:53.684364 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260928 09:01:54.425169 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:54.424927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:57885)
0928 09:01:54.425056 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:54.425119 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 09:01:54.546510 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:54.546239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:53231)
0928 09:01:54.546400 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:54.546459 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260928 09:01:54.685173 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:54.684952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:51403)
0928 09:01:54.685082 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:54.685127 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 09:01:55.101832 15723 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:55.101976 15723 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:55.102192 15723 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
W20260928 09:01:55.102653 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
W20260928 09:01:55.102728 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:55.102744 15322 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83440db9bf8a4040b1a2335a65af3813; no voters: 6f23c0d3d1664e56beca6d00861a6477, 7962d529ebcc42e38d4db7542a9e17d7
I20260928 09:01:55.102843 15723 raft_consensus.cc:2726] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:01:55.425989 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:55.425750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:58181)
0928 09:01:55.425893 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:55.425941 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260928 09:01:55.547338 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:55.547092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:58745)
0928 09:01:55.547243 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:55.547290 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260928 09:01:55.685954 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:55.685684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:51019)
0928 09:01:55.685853 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:55.685902 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260928 09:01:56.426842 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:56.426558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:40345)
0928 09:01:56.426737 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:56.426795 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260928 09:01:56.548091 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:56.547842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:34407)
0928 09:01:56.547994 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:56.548043 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260928 09:01:56.686784 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:56.686534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:53317)
0928 09:01:56.686684 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:56.686734 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 09:01:56.874575 15730 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:01:56.874673 15730 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:01:56.874867 15730 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
W20260928 09:01:56.875380 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
W20260928 09:01:56.875494 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
I20260928 09:01:56.875535 15322 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83440db9bf8a4040b1a2335a65af3813; no voters: 6f23c0d3d1664e56beca6d00861a6477, 7962d529ebcc42e38d4db7542a9e17d7
I20260928 09:01:56.875643 15730 raft_consensus.cc:2726] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:01:57.427707 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:57.427370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:56435)
0928 09:01:57.427517 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:57.427658 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 09:01:57.548867 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:57.548596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:55909)
0928 09:01:57.548776 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:57.548819 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260928 09:01:57.687558 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:57.687294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:55839)
0928 09:01:57.687433 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:57.687512 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260928 09:01:58.428438 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:58.428244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:39357)
0928 09:01:58.428362 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:58.428392 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260928 09:01:58.549593 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:58.549333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:49371)
0928 09:01:58.549475 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:58.549522 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260928 09:01:58.688279 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:58.688054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:55181)
0928 09:01:58.688191 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:58.688234 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260928 09:01:59.429322 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:59.429040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:40523)
0928 09:01:59.429165 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:59.429214 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 09:01:59.550370 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:59.550137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:41075)
0928 09:01:59.550283 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:59.550325 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260928 09:01:59.689069 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:01:59.688856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:41331)
0928 09:01:59.688980 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:01:59.689023 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 09:02:00.430297 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:00.429977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:54695)
0928 09:02:00.430200 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:00.430247 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260928 09:02:00.551047 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:00.550807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:58205)
0928 09:02:00.550957 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:00.550997 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260928 09:02:00.689802 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:00.689549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:52131)
0928 09:02:00.689711 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:00.689754 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260928 09:02:01.431037 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:01.430793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:56447)
0928 09:02:01.430953 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:01.430988 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260928 09:02:01.551785 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:01.551545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:45311)
0928 09:02:01.551694 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:01.551737 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260928 09:02:01.690618 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:01.690311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:53335)
0928 09:02:01.690515 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:01.690561 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":61,"threads_started":1}
I20260928 09:02:01.694478 15746 raft_consensus.cc:482] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:01.694591 15746 raft_consensus.cc:504] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f23c0d3d1664e56beca6d00861a6477" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35735 } } peers { permanent_uuid: "7962d529ebcc42e38d4db7542a9e17d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33381 } } peers { permanent_uuid: "83440db9bf8a4040b1a2335a65af3813" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36993 } }
I20260928 09:02:01.694805 15746 leader_election.cc:288] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735), 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381)
W20260928 09:02:01.695268 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7962d529ebcc42e38d4db7542a9e17d7 (127.0.98.3:33381): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
W20260928 09:02:01.695343 15322 leader_election.cc:341] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f23c0d3d1664e56beca6d00861a6477 (127.0.98.2:35735): Not found: Tablet not found: 945876ba534b43199d9bbe8cffec2e8c
I20260928 09:02:01.695365 15322 leader_election.cc:302] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83440db9bf8a4040b1a2335a65af3813; no voters: 6f23c0d3d1664e56beca6d00861a6477, 7962d529ebcc42e38d4db7542a9e17d7
I20260928 09:02:01.695448 15746 raft_consensus.cc:2726] T 945876ba534b43199d9bbe8cffec2e8c P 83440db9bf8a4040b1a2335a65af3813 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:02:02.431802 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:02.431555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:46201)
0928 09:02:02.431710 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:02.431753 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260928 09:02:02.552574 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:02.552333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:33001)
0928 09:02:02.552478 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:02.552527 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260928 09:02:02.691403 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:02.691184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:57789)
0928 09:02:02.691316 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:02.691359 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260928 09:02:03.432719 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:03.432412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:53103)
0928 09:02:03.432562 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:03.432616 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260928 09:02:03.553416 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:03.553173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:53997)
0928 09:02:03.553321 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:03.553367 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260928 09:02:03.692272 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:03.691930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:40111)
0928 09:02:03.692129 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:03.692210 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":109,"threads_started":1}
I20260928 09:02:04.433612 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:04.433371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.1:51893)
0928 09:02:04.433514 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:04.433564 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260928 09:02:04.554203 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:04.553966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.2:56823)
0928 09:02:04.554088 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:04.554157 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 09:02:04.688683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15274
I20260928 09:02:04.693171 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:04.692915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42083 (local address 127.0.98.3:51335)
0928 09:02:04.693082 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:04.693125 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260928 09:02:04.693838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15438
I20260928 09:02:04.698383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15570
2026-09-28T09:02:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18243 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:02:04 dist-test-slave-286n krb5kdc[15758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:02:04 dist-test-slave-286n krb5kdc[15758](info): set up 2 sockets
Sep 28 09:02:04 dist-test-slave-286n krb5kdc[15758](info): commencing operation
krb5kdc: starting...
W20260928 09:02:05.740921 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:02:05 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:02:05.863042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:05.944454 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:05.944705 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:05.944743 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:05.945953 15771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 09:02:05.946020 15771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 09:02:05.946043 15771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 09:02:05.946062 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:02:05.946097 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:05.946123 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:05.946151 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:02:05.946178 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:02:05.947705 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:05.947981 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:05.948220 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:05.953627 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:05.953629 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:05.953703 15779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:05.953981 15771 server_base.cc:1034] running on GCE node
I20260928 09:02:05.954229 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:05.954478 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:05.955622 15771 hybrid_clock.cc:648] HybridClock initialized: now 1790586125955608 us; error 32 us; skew 500 ppm
Sep 28 09:02:05 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:05.958266 15771 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:02:05.958820 15771 webserver.cc:460] Webserver started at http://127.0.98.62:39451/ using document root <none> and password file <none>
I20260928 09:02:05.959117 15771 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:05.959170 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:05.959349 15771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:02:05.960570 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "c1767d174a174562888c7e2e54eadaba"
format_stamp: "Formatted at 2026-09-28 09:02:05 on dist-test-slave-286n"
server_key: "4fcc47bdb42c572f7961381281f398fd"
server_key_iv: "47c57f478227afc971d29c460812d9d4"
server_key_version: "encryptionkey@0"
I20260928 09:02:05.961050 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1767d174a174562888c7e2e54eadaba"
format_stamp: "Formatted at 2026-09-28 09:02:05 on dist-test-slave-286n"
server_key: "4fcc47bdb42c572f7961381281f398fd"
server_key_iv: "47c57f478227afc971d29c460812d9d4"
server_key_version: "encryptionkey@0"
I20260928 09:02:05.962576 15771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:02:05.963567 15786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:05.963833 15771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:02:05.963958 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "c1767d174a174562888c7e2e54eadaba"
format_stamp: "Formatted at 2026-09-28 09:02:05 on dist-test-slave-286n"
server_key: "4fcc47bdb42c572f7961381281f398fd"
server_key_iv: "47c57f478227afc971d29c460812d9d4"
server_key_version: "encryptionkey@0"
I20260928 09:02:05.964046 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:05.999737 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:06.001868 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:06.002007 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:06.006294 15771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36365
I20260928 09:02:06.006362 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36365 every 8 connection(s)
I20260928 09:02:06.006731 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:02:06.007683 15839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:06.008316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15771
I20260928 09:02:06.008419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:02:06.008627 392 external_mini_cluster.cc:1428] Setting key 65e66d979e067d05534b1238abd9b2d7
I20260928 09:02:06.010950 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap starting.
I20260928 09:02:06.011547 15839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:06.011843 15839 log.cc:824] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:06.012571 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: No bootstrap required, opened a new log
I20260928 09:02:06.014606 15839 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:06.014745 15839 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:06.014768 15839 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1767d174a174562888c7e2e54eadaba, State: Initialized, Role: FOLLOWER
I20260928 09:02:06.014914 15839 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:06.015023 15839 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:02:06.015057 15839 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:02:06.015108 15839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 0 FOLLOWER]: Advancing to term 1
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:06.015869 15839 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:06.015991 15839 leader_election.cc:302] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1767d174a174562888c7e2e54eadaba; no voters:
I20260928 09:02:06.016178 15839 leader_election.cc:288] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:02:06.016207 15844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:02:06.016331 15844 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 LEADER]: Becoming Leader. State: Replica: c1767d174a174562888c7e2e54eadaba, State: Running, Role: LEADER
I20260928 09:02:06.016477 15839 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:02:06.016480 15844 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:06.017141 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1767d174a174562888c7e2e54eadaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260928 09:02:06.017230 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: This master's current role is: LEADER
I20260928 09:02:06.017834 15846 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1767d174a174562888c7e2e54eadaba. Latest consensus state: current_term: 1 leader_uuid: "c1767d174a174562888c7e2e54eadaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
W20260928 09:02:06.018376 15860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:02:06.018428 15860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:02:06.018481 15853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:02:06.018682 15853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:02:06.018126 15846 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: This master's current role is: LEADER
I20260928 09:02:06.020277 15853 catalog_manager.cc:1347] Generated new cluster ID: 3181d671337b492c848601714f4f3237
I20260928 09:02:06.020330 15853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:02:06.020550 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.009187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.0.98.62:36365)
0928 09:02:06.009410 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.009418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.009444 (+ 26us) server_negotiation.cc:407] Connection header received
0928 09:02:06.009904 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.009911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.009958 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.010076 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:06.010874 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.011879 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.012876 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.013292 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.013413 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.016405 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:06.016409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:06.016410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:06.016422 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:06.018715 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.019138 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.019141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.019142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.019265 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.019479 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.019480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.019480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.019785 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:06.019956 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.020197 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.020361 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 09:02:06.062456 15853 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:02:06.063107 15853 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:02:06.073818 15853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Generated new TSK 0
I20260928 09:02:06.074069 15853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:02:06.112702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36365
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:06.198515 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:06.198858 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:06.198910 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:06.200608 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:06.200693 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:06.200738 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:02:06.202975 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:06.203255 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:06.203495 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:06.209013 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:06.209030 15873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:06.209065 15871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:06.209375 15865 server_base.cc:1034] running on GCE node
I20260928 09:02:06.209530 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:06.209723 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:06.210877 15865 hybrid_clock.cc:648] HybridClock initialized: now 1790586126210862 us; error 28 us; skew 500 ppm
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:06.213799 15865 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:02:06.214394 15865 webserver.cc:460] Webserver started at http://127.0.98.1:35481/ using document root <none> and password file <none>
I20260928 09:02:06.214658 15865 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:06.214735 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:06.214895 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:02:06.216077 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a0f8e02746848d3813c0ad1df215e7b"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "03e00b5434f12362e96f25fdf8b70726"
server_key_iv: "d72c9527838188e3ba99da189a83b3a0"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.216475 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a0f8e02746848d3813c0ad1df215e7b"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "03e00b5434f12362e96f25fdf8b70726"
server_key_iv: "d72c9527838188e3ba99da189a83b3a0"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.218127 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:06.219067 15880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.219288 15865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:06.219414 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "6a0f8e02746848d3813c0ad1df215e7b"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "03e00b5434f12362e96f25fdf8b70726"
server_key_iv: "d72c9527838188e3ba99da189a83b3a0"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.219497 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:06.253932 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:06.256215 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:06.256388 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:06.256672 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:06.256997 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:06.257052 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.257093 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:06.257145 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.263458 15865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41271
I20260928 09:02:06.263502 15993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41271 every 8 connection(s)
I20260928 09:02:06.263825 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:02:06.268393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15865
I20260928 09:02:06.268484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:02:06.268741 392 external_mini_cluster.cc:1428] Setting key 29ca217e1edb0948c3450fd7d29d2d0c
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:06.274479 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.264561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52053 (local address 127.0.98.62:36365)
0928 09:02:06.264648 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.264672 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.264908 (+ 236us) server_negotiation.cc:407] Connection header received
0928 09:02:06.265515 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.265516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.265531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.265603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:06.266613 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.267206 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.268342 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.268698 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.268776 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.271200 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:06.271202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:06.271203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:06.271211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:06.272824 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.273438 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.273439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.273439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.273489 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.273788 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.273788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.273789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.273893 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:06.273943 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.274334 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.274412 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 09:02:06.274816 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.264472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:52053)
0928 09:02:06.264730 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.264803 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.265390 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.265687 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.265692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.266287 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:06.266494 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.266500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.267309 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.267310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.268241 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.268243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.268787 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.268789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.268963 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.269606 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:06.269615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:06.271092 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:06.272908 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:06.272916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:06.272919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:06.273350 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:06.273554 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:06.273554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:06.273555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:06.273719 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:06.273962 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:06.273969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:06.274164 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.274312 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.274514 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
I20260928 09:02:06.275431 15994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36365
I20260928 09:02:06.275540 15994 heartbeater.cc:464] Registering TS with master...
I20260928 09:02:06.275751 15994 heartbeater.cc:511] Master 127.0.98.62:36365 requested a full tablet report, sending...
I20260928 09:02:06.276364 15803 ts_manager.cc:194] Registered new tserver with Master: 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271)
I20260928 09:02:06.277091 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52053
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:02:06.296406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36365
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:06.379210 15999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:06.379446 15999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:06.379472 15999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:06.380733 15999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:06.380784 15999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:06.380813 15999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:02:06.382356 15999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:06.382620 15999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:06.382897 15999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:06.388134 16005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:06.388216 16004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:06.388134 16007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:06.388412 15999 server_base.cc:1034] running on GCE node
I20260928 09:02:06.388794 15999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:06.389068 15999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:06.390239 15999 hybrid_clock.cc:648] HybridClock initialized: now 1790586126390230 us; error 31 us; skew 500 ppm
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:06.392978 15999 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:02:06.393456 15999 webserver.cc:460] Webserver started at http://127.0.98.2:44199/ using document root <none> and password file <none>
I20260928 09:02:06.393648 15999 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:06.393702 15999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:06.393795 15999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:02:06.394798 15999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "d77a80d27e3a49789f85e0bce66bc924"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "cef1e500c053b2976976183cae88dc4c"
server_key_iv: "739b32ae55bfe1dc0df1f57a222bb652"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.395093 15999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d77a80d27e3a49789f85e0bce66bc924"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "cef1e500c053b2976976183cae88dc4c"
server_key_iv: "739b32ae55bfe1dc0df1f57a222bb652"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.396405 15999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:02:06.397246 16014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.397400 15999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.397485 15999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "d77a80d27e3a49789f85e0bce66bc924"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "cef1e500c053b2976976183cae88dc4c"
server_key_iv: "739b32ae55bfe1dc0df1f57a222bb652"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.397541 15999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:06.413548 15999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:06.415813 15999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:06.415961 15999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:06.416183 15999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:06.416626 15999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:06.416728 15999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.416781 15999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:06.416814 15999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.423625 15999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35561
I20260928 09:02:06.423659 16127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35561 every 8 connection(s)
I20260928 09:02:06.424041 15999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:06.431629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 15999
I20260928 09:02:06.431754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:02:06.432001 392 external_mini_cluster.cc:1428] Setting key e4dbcf2aea7998bd435c321684a2f666
I20260928 09:02:06.434242 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.424881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45177 (local address 127.0.98.62:36365)
0928 09:02:06.424966 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.424969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.425171 (+ 202us) server_negotiation.cc:407] Connection header received
0928 09:02:06.425632 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.425633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.425644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.425709 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:06.426678 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.427222 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.428330 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.428526 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.428594 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.430867 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:06.430869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:06.430869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:06.430876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:06.432713 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.433171 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.433172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.433172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.433235 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.433475 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.433476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.433476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.433642 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:06.433710 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.433983 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.434076 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 09:02:06.434473 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.424856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:45177)
0928 09:02:06.425035 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.425071 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.425538 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.425760 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.425763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.426382 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:06.426573 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.426578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.427341 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.427342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.428238 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.428241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.428617 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.428618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.428751 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.429298 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:06.429308 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:06.430767 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:06.432820 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:06.432829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:06.432832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:06.433095 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:06.433302 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:06.433302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:06.433303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:06.433410 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:06.433710 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:06.433716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:06.433852 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.433939 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.434070 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260928 09:02:06.434965 16128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36365
I20260928 09:02:06.435078 16128 heartbeater.cc:464] Registering TS with master...
I20260928 09:02:06.435299 16128 heartbeater.cc:511] Master 127.0.98.62:36365 requested a full tablet report, sending...
I20260928 09:02:06.435756 15803 ts_manager.cc:194] Registered new tserver with Master: d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561)
I20260928 09:02:06.436139 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45177
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:02:06.459877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36365
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:06.545252 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:06.545459 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:06.545482 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:06.546725 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:06.546789 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:06.546818 16133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:02:06.548230 16133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:06.548412 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:06.548599 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:06.553777 16139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:06.553768 16138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:06.553768 16141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:06.553890 16133 server_base.cc:1034] running on GCE node
I20260928 09:02:06.554204 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:06.554473 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:06.555652 16133 hybrid_clock.cc:648] HybridClock initialized: now 1790586126555639 us; error 36 us; skew 500 ppm
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:06.558233 16133 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:02:06.558700 16133 webserver.cc:460] Webserver started at http://127.0.98.3:36813/ using document root <none> and password file <none>
I20260928 09:02:06.558923 16133 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:06.558969 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:06.559110 16133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:02:06.560190 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "3691fbc148f64cff8b035867384c3849"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "17feb4ea2acde007c226fe2c8af3b6d9"
server_key_iv: "a9a6598983615120acf46b283957c570"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.560557 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3691fbc148f64cff8b035867384c3849"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "17feb4ea2acde007c226fe2c8af3b6d9"
server_key_iv: "a9a6598983615120acf46b283957c570"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.561918 16133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:06.562784 16148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.562965 16133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:02:06.563076 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "3691fbc148f64cff8b035867384c3849"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "17feb4ea2acde007c226fe2c8af3b6d9"
server_key_iv: "a9a6598983615120acf46b283957c570"
server_key_version: "encryptionkey@0"
I20260928 09:02:06.563176 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:06.575909 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:06.578162 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:06.578339 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:06.578598 16133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:06.578928 16133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:06.578987 16133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.579026 16133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:06.579078 16133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:06.585734 16133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42339
I20260928 09:02:06.585769 16261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42339 every 8 connection(s)
I20260928 09:02:06.586125 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 09:02:06 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:06.595321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16133
I20260928 09:02:06.595436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:02:06.595654 392 external_mini_cluster.cc:1428] Setting key 3dd49ec000e7ca2de80cd406a0d99cf3
I20260928 09:02:06.596127 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.586924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44817 (local address 127.0.98.62:36365)
0928 09:02:06.586988 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.586990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.587245 (+ 255us) server_negotiation.cc:407] Connection header received
0928 09:02:06.587758 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.587758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.587769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.587832 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:06.588943 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.589463 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.590490 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.590682 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.590735 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.593142 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:06.593144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:06.593144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:06.593152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:06.594499 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.594897 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.594898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.594899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.594955 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.595237 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.595238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.595239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.595403 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:06.595499 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.595896 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.596013 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:02:06.596345 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.586891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:44817)
0928 09:02:06.587080 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.587124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.587660 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.587908 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.587911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.588569 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:06.588852 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.588858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.589564 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.589565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.590410 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.590413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.590757 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.590758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.590856 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.591333 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:06.591341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:06.593054 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:06.594589 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:06.594595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:06.594597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:06.594825 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:06.595037 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:06.595039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:06.595039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:06.595155 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:06.595533 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:06.595540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:06.595719 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.595849 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.596052 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":116,"threads_started":1}
I20260928 09:02:06.596832 16262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36365
I20260928 09:02:06.596953 16262 heartbeater.cc:464] Registering TS with master...
I20260928 09:02:06.597142 16262 heartbeater.cc:511] Master 127.0.98.62:36365 requested a full tablet report, sending...
I20260928 09:02:06.597569 15803 ts_manager.cc:194] Registered new tserver with Master: 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339)
I20260928 09:02:06.597955 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44817
I20260928 09:02:06.607758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:02:06.615420 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.609293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.0.98.62:36365)
0928 09:02:06.609473 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.609475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.609484 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:02:06.609505 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.609506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.609515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.609563 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:06.610101 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.610615 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.611377 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.611560 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.611623 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.612869 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:06.612870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:06.612870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:06.612876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:06.614312 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.614646 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.614647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.614647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.614699 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:06.614917 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:06.614918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:06.614918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:06.615102 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:06.615184 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.615269 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.615332 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 09:02:06.617112 15803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:02:06.618862 15803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:02:06.626454 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.621328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41271 (local address 127.0.0.1:58768)
0928 09:02:06.621780 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.621788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.621832 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.622259 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.622260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.622262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:06.622353 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.622355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.623753 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.623754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.625234 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.625236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.626075 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.626118 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.626199 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.626213 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.626305 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.626368 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":80,"thread_start_us":72,"threads_started":1}
I20260928 09:02:06.626749 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.621328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35561 (local address 127.0.0.1:59896)
0928 09:02:06.621581 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.621592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.621646 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.622308 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.622308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.622310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:06.622355 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.622356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.624762 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.624763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.625616 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.625619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.626486 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.626487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.626553 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.626557 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.626634 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.626673 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260928 09:02:06.626825 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.621603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.0.98.1:41271)
0928 09:02:06.621753 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.621758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.621830 (+ 72us) server_negotiation.cc:407] Connection header received
0928 09:02:06.621889 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.621892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.621908 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.621991 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:06.622460 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.623642 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.625329 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.625974 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.626040 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.626082 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.626674 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.626733 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260928 09:02:06.627244 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.621320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42339 (local address 127.0.0.1:49196)
0928 09:02:06.621422 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.621460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.621555 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.622241 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.622245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.622247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:06.622348 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.622349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.623755 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.623755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.625234 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.625236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.627030 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.627031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.627096 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.627102 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.627129 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.627169 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":108,"thread_start_us":59,"threads_started":1}
I20260928 09:02:06.627573 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.621599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.98.3:42339)
0928 09:02:06.622011 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.622025 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.622034 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:02:06.622071 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.622074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.622089 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.622217 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:06.622428 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.623642 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.626113 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.626921 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.627366 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.627437 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.627462 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.627511 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":91,"threads_started":1}
I20260928 09:02:06.627478 15928 tablet_service.cc:1467] Processing CreateTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 (DEFAULT_TABLE table=test-table [id=c09f490ebdeb46d0bb14a8b9652b546e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:02:06.627858 15928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46bea28bf3c473a9f5ce67f69284c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:06.628219 16196 tablet_service.cc:1467] Processing CreateTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 (DEFAULT_TABLE table=test-table [id=c09f490ebdeb46d0bb14a8b9652b546e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:02:06.628525 16196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46bea28bf3c473a9f5ce67f69284c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:06.627038 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.621800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.2:35561)
0928 09:02:06.621996 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.622001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.622010 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:02:06.622052 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.622054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.622069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.622187 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:06.623772 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.624621 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.625732 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.626344 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.626403 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.626453 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.626914 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.626959 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260928 09:02:06.630497 16279 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap starting.
I20260928 09:02:06.631162 16279 tablet_bootstrap.cc:654] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:06.631181 16062 tablet_service.cc:1467] Processing CreateTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 (DEFAULT_TABLE table=test-table [id=c09f490ebdeb46d0bb14a8b9652b546e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:02:06.631474 16280 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Bootstrap starting.
I20260928 09:02:06.631549 16062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f46bea28bf3c473a9f5ce67f69284c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:06.631773 16279 log.cc:824] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:06.632114 16280 tablet_bootstrap.cc:654] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:06.632663 16280 log.cc:824] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:06.632679 16279 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: No bootstrap required, opened a new log
I20260928 09:02:06.632750 16279 ts_tablet_manager.cc:1397] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:02:06.633419 16280 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: No bootstrap required, opened a new log
I20260928 09:02:06.633476 16280 ts_tablet_manager.cc:1397] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:02:06.634191 16279 raft_consensus.cc:348] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.634315 16279 raft_consensus.cc:374] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:06.634335 16279 raft_consensus.cc:729] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0f8e02746848d3813c0ad1df215e7b, State: Initialized, Role: FOLLOWER
I20260928 09:02:06.634454 16279 consensus_queue.cc:260] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.634737 15994 heartbeater.cc:503] Master 127.0.98.62:36365 was elected leader, sending a full tablet report...
I20260928 09:02:06.634739 16279 ts_tablet_manager.cc:1428] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:02:06.635242 16284 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Bootstrap starting.
I20260928 09:02:06.635469 16280 raft_consensus.cc:348] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.635670 16280 raft_consensus.cc:374] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:06.635704 16280 raft_consensus.cc:729] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3691fbc148f64cff8b035867384c3849, State: Initialized, Role: FOLLOWER
I20260928 09:02:06.635871 16284 tablet_bootstrap.cc:654] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:06.635834 16280 consensus_queue.cc:260] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.636138 16280 ts_tablet_manager.cc:1428] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:02:06.636165 16262 heartbeater.cc:503] Master 127.0.98.62:36365 was elected leader, sending a full tablet report...
I20260928 09:02:06.636422 16284 log.cc:824] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:06.637149 16284 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: No bootstrap required, opened a new log
I20260928 09:02:06.637204 16284 ts_tablet_manager.cc:1397] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:02:06.638522 16284 raft_consensus.cc:348] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.638661 16284 raft_consensus.cc:374] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:06.638684 16284 raft_consensus.cc:729] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77a80d27e3a49789f85e0bce66bc924, State: Initialized, Role: FOLLOWER
I20260928 09:02:06.638810 16284 consensus_queue.cc:260] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.639073 16284 ts_tablet_manager.cc:1428] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:02:06.639086 16128 heartbeater.cc:503] Master 127.0.98.62:36365 was elected leader, sending a full tablet report...
W20260928 09:02:06.674671 16129 tablet.cc:2369] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:02:06.764509 15995 tablet.cc:2369] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:02:06.836817 16263 tablet.cc:2369] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:02:06.853698 16287 raft_consensus.cc:482] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:06.853845 16287 raft_consensus.cc:504] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.854213 16287 leader_election.cc:288] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339), 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271)
I20260928 09:02:06.858608 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.854569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47893 (local address 127.0.98.3:42339)
0928 09:02:06.854632 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.854634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.854644 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:02:06.854684 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.854684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.854695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.854816 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:06.855050 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.855929 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.857516 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.858152 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.858230 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.858251 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.858486 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.858534 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:02:06.858614 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.854511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41489 (local address 127.0.98.1:41271)
0928 09:02:06.854605 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.854608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.854734 (+ 126us) server_negotiation.cc:407] Connection header received
0928 09:02:06.854766 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.854767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.854782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.854852 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:06.855040 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.855888 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.857517 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.858131 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.858230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.858245 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.858486 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.858534 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 09:02:06.858594 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.854441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42339 (local address 127.0.98.2:47893)
0928 09:02:06.854487 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.854499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.854557 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.854811 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.854812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.854813 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:06.854941 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.854942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.856039 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.856041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.857398 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.857401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.858271 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.858272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.858359 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.858361 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.858429 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.858484 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":56}
I20260928 09:02:06.858601 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.854441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41271 (local address 127.0.98.2:41489)
0928 09:02:06.854665 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.854672 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:06.854711 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:06.854863 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:06.854864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:06.854865 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:06.854924 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.854927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.856039 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.856041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.857406 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:06.857407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.858271 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.858272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:06.858356 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.858360 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:02:06.858434 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:02:06.858484 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":134,"thread_start_us":108,"threads_started":1}
I20260928 09:02:06.859000 16216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46bea28bf3c473a9f5ce67f69284c80" candidate_uuid: "d77a80d27e3a49789f85e0bce66bc924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3691fbc148f64cff8b035867384c3849" is_pre_election: true
I20260928 09:02:06.859030 15948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46bea28bf3c473a9f5ce67f69284c80" candidate_uuid: "d77a80d27e3a49789f85e0bce66bc924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0f8e02746848d3813c0ad1df215e7b" is_pre_election: true
I20260928 09:02:06.859184 16216 raft_consensus.cc:2454] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d77a80d27e3a49789f85e0bce66bc924 in term 0.
I20260928 09:02:06.859184 15948 raft_consensus.cc:2454] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d77a80d27e3a49789f85e0bce66bc924 in term 0.
I20260928 09:02:06.859453 16016 leader_election.cc:302] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0f8e02746848d3813c0ad1df215e7b, d77a80d27e3a49789f85e0bce66bc924; no voters:
I20260928 09:02:06.859643 16287 raft_consensus.cc:2781] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:02:06.859758 16287 raft_consensus.cc:482] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:02:06.859797 16287 raft_consensus.cc:3037] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:02:06.860716 16287 raft_consensus.cc:504] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.860925 16287 leader_election.cc:288] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [CANDIDATE]: Term 1 election: Requested vote from peers 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339), 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271)
I20260928 09:02:06.861159 16216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46bea28bf3c473a9f5ce67f69284c80" candidate_uuid: "d77a80d27e3a49789f85e0bce66bc924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3691fbc148f64cff8b035867384c3849"
I20260928 09:02:06.861215 15948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f46bea28bf3c473a9f5ce67f69284c80" candidate_uuid: "d77a80d27e3a49789f85e0bce66bc924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0f8e02746848d3813c0ad1df215e7b"
I20260928 09:02:06.861258 16216 raft_consensus.cc:3037] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:02:06.861282 15948 raft_consensus.cc:3037] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:02:06.862149 16216 raft_consensus.cc:2454] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d77a80d27e3a49789f85e0bce66bc924 in term 1.
I20260928 09:02:06.862200 15948 raft_consensus.cc:2454] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d77a80d27e3a49789f85e0bce66bc924 in term 1.
I20260928 09:02:06.862388 16018 leader_election.cc:302] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924; no voters:
I20260928 09:02:06.862528 16287 raft_consensus.cc:2781] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:02:06.862737 16287 raft_consensus.cc:686] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 1 LEADER]: Becoming Leader. State: Replica: d77a80d27e3a49789f85e0bce66bc924, State: Running, Role: LEADER
I20260928 09:02:06.862906 16287 consensus_queue.cc:237] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:06.863888 15803 catalog_manager.cc:5495] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 reported cstate change: term changed from 0 to 1, leader changed from <none> to d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d77a80d27e3a49789f85e0bce66bc924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } health_report { overall_health: UNKNOWN } } }
I20260928 09:02:06.909456 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.906789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.0.98.2:35561)
0928 09:02:06.906850 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:06.906852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:06.906860 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:02:06.906931 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:06.906932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:06.906956 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:06.907021 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:02:06.907151 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.907696 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.908428 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:06.908612 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:06.908715 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:06.909008 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:02:06.909128 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:02:06.909208 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:06.909297 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 09:02:06.909317 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:02:06.911440 16216 raft_consensus.cc:1261] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Refusing update from remote peer d77a80d27e3a49789f85e0bce66bc924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:02:06.911433 15948 raft_consensus.cc:1261] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Refusing update from remote peer d77a80d27e3a49789f85e0bce66bc924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:02:06.911906 16289 consensus_queue.cc:1035] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:02:06.912060 16287 consensus_queue.cc:1035] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:02:06.914417 16292 mvcc.cc:205] Tried to move back new op lower bound from 7334240775827120128 to 7334240775632158720. Current Snapshot: MvccSnapshot[applied={T|T < 7334240775827120128}]
I20260928 09:02:06.915390 16293 mvcc.cc:205] Tried to move back new op lower bound from 7334240775827120128 to 7334240775632158720. Current Snapshot: MvccSnapshot[applied={T|T < 7334240775827120128}]
I20260928 09:02:06.915227 16294 mvcc.cc:205] Tried to move back new op lower bound from 7334240775827120128 to 7334240775632158720. Current Snapshot: MvccSnapshot[applied={T|T < 7334240775827120128 or (T in {7334240775827120128})}]
I20260928 09:02:06.916790 15801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:02:06.916888 15801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:02:06.917657 15801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Sending DeleteTablet for 3 replicas of tablet f46bea28bf3c473a9f5ce67f69284c80
I20260928 09:02:06.917994 16196 tablet_service.cc:1514] Processing DeleteTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49196
I20260928 09:02:06.918031 15928 tablet_service.cc:1514] Processing DeleteTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58768
I20260928 09:02:06.918212 16062 tablet_service.cc:1514] Processing DeleteTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59896
I20260928 09:02:06.918253 16302 tablet_replica.cc:330] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: stopping tablet replica
I20260928 09:02:06.918357 16302 raft_consensus.cc:2229] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:02:06.918403 16304 tablet_replica.cc:330] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: stopping tablet replica
I20260928 09:02:06.918470 16302 raft_consensus.cc:2258] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:02:06.918509 16304 raft_consensus.cc:2229] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:02:06.918650 16304 raft_consensus.cc:2258] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:02:06.918689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15865
I20260928 09:02:06.918973 16302 ts_tablet_manager.cc:1905] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:02:06.919091 16304 ts_tablet_manager.cc:1905] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:02:06.920792 16302 ts_tablet_manager.cc:1918] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:02:06.920863 16302 log.cc:1196] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/f46bea28bf3c473a9f5ce67f69284c80
I20260928 09:02:06.921101 16304 ts_tablet_manager.cc:1918] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:02:06.921162 16302 ts_tablet_manager.cc:1939] T f46bea28bf3c473a9f5ce67f69284c80 P 3691fbc148f64cff8b035867384c3849: Deleting consensus metadata
I20260928 09:02:06.921165 16304 log.cc:1196] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/f46bea28bf3c473a9f5ce67f69284c80
I20260928 09:02:06.921409 16304 ts_tablet_manager.cc:1939] T f46bea28bf3c473a9f5ce67f69284c80 P d77a80d27e3a49789f85e0bce66bc924: Deleting consensus metadata
I20260928 09:02:06.922053 15787 catalog_manager.cc:4841] TS d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561): tablet f46bea28bf3c473a9f5ce67f69284c80 (table test-table [id=c09f490ebdeb46d0bb14a8b9652b546e]) successfully deleted
I20260928 09:02:06.922466 15790 catalog_manager.cc:4841] TS 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339): tablet f46bea28bf3c473a9f5ce67f69284c80 (table test-table [id=c09f490ebdeb46d0bb14a8b9652b546e]) successfully deleted
W20260928 09:02:06.925812 15788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41271 (error 108)
I20260928 09:02:06.926033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15999
I20260928 09:02:06.926606 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:06.926243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41271 (local address 127.0.0.1:58780)
0928 09:02:06.926522 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:06.926558 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258}
W20260928 09:02:06.926820 15788 catalog_manager.cc:4584] TS 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f46bea28bf3c473a9f5ce67f69284c80: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41271: connect: Connection refused (error 111)
I20260928 09:02:06.931598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 16133
I20260928 09:02:06.935853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 15771
I20260928 09:02:26.941498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--webserver_interface=127.0.98.62
--webserver_port=39451
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:27.022328 16306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:27.022611 16306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:27.022652 16306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:27.023895 16306 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 09:02:27.023964 16306 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 09:02:27.023990 16306 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 09:02:27.024010 16306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:02:27.024037 16306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:27.024071 16306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:27.024096 16306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:02:27.024125 16306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:02:27.025784 16306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:27.026094 16306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:27.026329 16306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:27.035267 16312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.035269 16313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.035254 16315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:27.035569 16306 server_base.cc:1034] running on GCE node
I20260928 09:02:27.035836 16306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:27.036165 16306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:27.037322 16306 hybrid_clock.cc:648] HybridClock initialized: now 1790586147037304 us; error 35 us; skew 500 ppm
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:27.048089 16306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:02:27.048697 16306 webserver.cc:460] Webserver started at http://127.0.98.62:39451/ using document root <none> and password file <none>
I20260928 09:02:27.048955 16306 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:27.049031 16306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:27.050521 16306 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:02:27.051230 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.051403 16306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:02:27.051522 16306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "c1767d174a174562888c7e2e54eadaba"
format_stamp: "Formatted at 2026-09-28 09:02:05 on dist-test-slave-286n"
server_key: "4fcc47bdb42c572f7961381281f398fd"
server_key_iv: "47c57f478227afc971d29c460812d9d4"
server_key_version: "encryptionkey@0"
I20260928 09:02:27.051959 16306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:27.063102 16306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:27.065230 16306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:27.065402 16306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:27.069810 16306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36365
I20260928 09:02:27.069847 16374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36365 every 8 connection(s)
I20260928 09:02:27.070279 16306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:02:27.071813 16375 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:02:27.072448 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap starting.
I20260928 09:02:27.076884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16306
I20260928 09:02:27.077204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41271
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36365
--webserver_port=35481
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:02:27.081517 16375 log.cc:824] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:27.083720 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:02:27.084054 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap complete.
I20260928 09:02:27.086459 16375 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:27.086688 16375 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1767d174a174562888c7e2e54eadaba, State: Initialized, Role: FOLLOWER
I20260928 09:02:27.086858 16375 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:27.086936 16375 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:02:27.087000 16375 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:02:27.087050 16375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 1 FOLLOWER]: Advancing to term 2
I20260928 09:02:27.088107 16375 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:27.088198 16375 leader_election.cc:302] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1767d174a174562888c7e2e54eadaba; no voters:
I20260928 09:02:27.088452 16375 leader_election.cc:288] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 09:02:27.088488 16379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Leader election won for term 2
I20260928 09:02:27.088603 16379 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 LEADER]: Becoming Leader. State: Replica: c1767d174a174562888c7e2e54eadaba, State: Running, Role: LEADER
I20260928 09:02:27.088766 16375 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:02:27.088792 16379 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:27.089118 16380 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1767d174a174562888c7e2e54eadaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260928 09:02:27.089291 16380 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: This master's current role is: LEADER
I20260928 09:02:27.089113 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1767d174a174562888c7e2e54eadaba. Latest consensus state: current_term: 2 leader_uuid: "c1767d174a174562888c7e2e54eadaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260928 09:02:27.089689 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: This master's current role is: LEADER
I20260928 09:02:27.089943 16388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:02:27.090833 16388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:02:27.091017 16388 catalog_manager.cc:1259] Loaded cluster ID: 3181d671337b492c848601714f4f3237
I20260928 09:02:27.091055 16388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:02:27.092175 16388 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:02:27.092360 16388 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Loaded TSK: 0
I20260928 09:02:27.107478 16388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Generated new TSK 1
I20260928 09:02:27.107615 16388 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:02:27.164956 16377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:27.165208 16377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:27.165248 16377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:27.166401 16377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:27.166468 16377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:27.166512 16377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:02:27.167953 16377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35481
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:27.168220 16377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:27.168453 16377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:27.173460 16403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.173460 16402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.173508 16405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:27.173763 16377 server_base.cc:1034] running on GCE node
I20260928 09:02:27.173950 16377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:27.174206 16377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:27.175366 16377 hybrid_clock.cc:648] HybridClock initialized: now 1790586147175339 us; error 46 us; skew 500 ppm
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:27.178164 16377 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:02:27.178697 16377 webserver.cc:460] Webserver started at http://127.0.98.1:35481/ using document root <none> and password file <none>
I20260928 09:02:27.178961 16377 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:27.179039 16377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:27.180868 16377 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:02:27.181629 16412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.181849 16377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:02:27.181947 16377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "6a0f8e02746848d3813c0ad1df215e7b"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "03e00b5434f12362e96f25fdf8b70726"
server_key_iv: "d72c9527838188e3ba99da189a83b3a0"
server_key_version: "encryptionkey@0"
I20260928 09:02:27.182421 16377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:27.207293 16377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:27.209951 16377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:27.210119 16377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:27.210400 16377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:27.210959 16419 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:02:27.212601 16377 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:02:27.212702 16377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:02:27.212740 16377 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:02:27.213600 16377 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:02:27.213649 16377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 09:02:27.213749 16419 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap starting.
I20260928 09:02:27.220901 16377 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41271
I20260928 09:02:27.221072 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41271 every 8 connection(s)
I20260928 09:02:27.221412 16377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:02:27.223007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16377
I20260928 09:02:27.223376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35561
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36365
--webserver_port=44199
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:02:27.226871 16419 log.cc:824] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:27.230171 16419 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:02:27.230556 16419 tablet_bootstrap.cc:492] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap complete.
I20260928 09:02:27.231954 16419 ts_tablet_manager.cc:1397] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260928 09:02:27.233855 16419 raft_consensus.cc:348] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.234261 16419 raft_consensus.cc:729] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0f8e02746848d3813c0ad1df215e7b, State: Initialized, Role: FOLLOWER
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:27.234666 16419 consensus_queue.cc:260] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.235085 16419 ts_tablet_manager.cc:1428] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 09:02:27.239316 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.222259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60955 (local address 127.0.98.62:36365)
0928 09:02:27.222492 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.222504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.227205 (+ 4701us) server_negotiation.cc:407] Connection header received
0928 09:02:27.227339 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.227345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.227398 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.227524 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:27.228497 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.229972 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.231478 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.232066 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.232162 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.235045 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:27.235050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:27.235051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:27.235064 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:27.237305 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.237854 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.237857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.237857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.237973 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.238279 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.238279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.238280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.238555 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:27.238636 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.238984 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.239113 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
I20260928 09:02:27.239337 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.222135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:60955)
0928 09:02:27.226266 (+ 4131us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.226316 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.226967 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.227655 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.227660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.228110 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:27.228361 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.228368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.230122 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.230123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.231381 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.231385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.232274 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.232276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.232429 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.233116 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:27.233127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:27.234929 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:27.237403 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:27.237411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:27.237413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:27.237752 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:27.238060 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:27.238061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:27.238061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:27.238204 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:27.238630 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:27.238636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:27.238803 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.238926 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.239109 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4089,"thread_start_us":90,"threads_started":1}
I20260928 09:02:27.240571 16527 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36365
I20260928 09:02:27.240694 16527 heartbeater.cc:464] Registering TS with master...
I20260928 09:02:27.240873 16527 heartbeater.cc:511] Master 127.0.98.62:36365 requested a full tablet report, sending...
I20260928 09:02:27.241647 16339 ts_manager.cc:194] Registered new tserver with Master: 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271)
I20260928 09:02:27.242986 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60955
I20260928 09:02:27.243939 16527 heartbeater.cc:503] Master 127.0.98.62:36365 was elected leader, sending a full tablet report...
I20260928 09:02:27.248574 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.242297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41271 (local address 127.0.0.1:59958)
0928 09:02:27.244739 (+ 2442us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.244777 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.244857 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.245063 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.245066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.245068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:27.245131 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.245133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.246502 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.246503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.247351 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.247354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.248339 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.248340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.248404 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.248414 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.248477 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.248517 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2411,"thread_start_us":101,"threads_started":1}
I20260928 09:02:27.248682 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.243314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.0.98.1:41271)
0928 09:02:27.243497 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.243502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.244861 (+ 1359us) server_negotiation.cc:407] Connection header received
0928 09:02:27.244949 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.244951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.244969 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.245051 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:27.245203 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.246415 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.247442 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.248244 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.248339 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.248397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.248513 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.248555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260928 09:02:27.249037 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet f46bea28bf3c473a9f5ce67f69284c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59958
I20260928 09:02:27.249575 16536 tablet_replica.cc:330] stopping tablet replica
I20260928 09:02:27.249684 16536 raft_consensus.cc:2229] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:02:27.249758 16536 raft_consensus.cc:2258] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:02:27.250069 16536 ts_tablet_manager.cc:1905] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:02:27.252223 16536 ts_tablet_manager.cc:1918] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:02:27.252302 16536 log.cc:1196] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal/wals/f46bea28bf3c473a9f5ce67f69284c80
I20260928 09:02:27.252529 16536 ts_tablet_manager.cc:1939] T f46bea28bf3c473a9f5ce67f69284c80 P 6a0f8e02746848d3813c0ad1df215e7b: Deleting consensus metadata
I20260928 09:02:27.253069 16324 catalog_manager.cc:4841] TS 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271): tablet f46bea28bf3c473a9f5ce67f69284c80 (table test-table [id=c09f490ebdeb46d0bb14a8b9652b546e]) successfully deleted
W20260928 09:02:27.322324 16531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:27.322592 16531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:27.322634 16531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:27.323932 16531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:27.324011 16531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:27.324064 16531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:02:27.325793 16531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44199
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:27.326089 16531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:27.326354 16531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:27.332084 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:27.332147 16531 server_base.cc:1034] running on GCE node
W20260928 09:02:27.332111 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.332094 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:27.332525 16531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:27.332829 16531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:27.334004 16531 hybrid_clock.cc:648] HybridClock initialized: now 1790586147333993 us; error 47 us; skew 500 ppm
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:27.336863 16531 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:02:27.337272 16531 webserver.cc:460] Webserver started at http://127.0.98.2:44199/ using document root <none> and password file <none>
I20260928 09:02:27.337456 16531 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:27.337509 16531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:27.338863 16531 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:27.339583 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.339761 16531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:02:27.339856 16531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "d77a80d27e3a49789f85e0bce66bc924"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "cef1e500c053b2976976183cae88dc4c"
server_key_iv: "739b32ae55bfe1dc0df1f57a222bb652"
server_key_version: "encryptionkey@0"
I20260928 09:02:27.340205 16531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:27.349210 16531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:27.351656 16531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:27.351810 16531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:27.352074 16531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:27.352420 16531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:27.352458 16531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.352481 16531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:27.352494 16531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.358386 16531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35561
I20260928 09:02:27.358474 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35561 every 8 connection(s)
I20260928 09:02:27.358825 16531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:02:27.359630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16531
I20260928 09:02:27.359943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42339
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36365
--webserver_port=36813
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:27.370703 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.359651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53935 (local address 127.0.98.62:36365)
0928 09:02:27.359716 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.359718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.360094 (+ 376us) server_negotiation.cc:407] Connection header received
0928 09:02:27.360965 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.360966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.360977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.361037 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:27.362192 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.362849 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.364518 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.364793 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.364863 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.367575 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:27.367576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:27.367576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:27.367583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:27.369141 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.369658 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.369659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.369659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.369711 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.370030 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.370031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.370031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.370143 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:27.370197 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.370559 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.370632 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:02:27.371057 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.359646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:53935)
0928 09:02:27.359947 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.359996 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.360847 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.361120 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.361124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.361862 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:27.362077 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.362082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.362976 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.362977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.364419 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.364422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.364887 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.364888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.365069 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.365634 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:27.365646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:27.367472 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:27.369224 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:27.369231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:27.369235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:27.369580 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:27.369778 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:27.369778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:27.369779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:27.369945 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:27.370214 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:27.370220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:27.370410 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.370544 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.370744 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260928 09:02:27.371650 16665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36365
I20260928 09:02:27.371754 16665 heartbeater.cc:464] Registering TS with master...
I20260928 09:02:27.372030 16665 heartbeater.cc:511] Master 127.0.98.62:36365 requested a full tablet report, sending...
I20260928 09:02:27.372396 16339 ts_manager.cc:194] Registered new tserver with Master: d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561)
I20260928 09:02:27.372838 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53935
W20260928 09:02:27.444124 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:27.444350 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:27.444398 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:27.445698 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:27.445784 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:27.445833 16668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:02:27.447319 16668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36813
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:27.447602 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:27.447831 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:02:27.452972 16668 server_base.cc:1034] running on GCE node
W20260928 09:02:27.452958 16674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.452945 16676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:27.452960 16673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:27.453424 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:27.453670 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:27.454833 16668 hybrid_clock.cc:648] HybridClock initialized: now 1790586147454824 us; error 32 us; skew 500 ppm
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:27.457516 16668 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:02:27.457962 16668 webserver.cc:460] Webserver started at http://127.0.98.3:36813/ using document root <none> and password file <none>
I20260928 09:02:27.458186 16668 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:27.458250 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:27.459741 16668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:02:27.460341 16683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.460523 16668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:02:27.460625 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "3691fbc148f64cff8b035867384c3849"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "17feb4ea2acde007c226fe2c8af3b6d9"
server_key_iv: "a9a6598983615120acf46b283957c570"
server_key_version: "encryptionkey@0"
I20260928 09:02:27.461102 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:27.473148 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:27.475983 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:27.476183 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:27.476483 16668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:27.476922 16668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:27.477078 16668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.477169 16668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:27.477234 16668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.483211 16668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42339
I20260928 09:02:27.483240 16796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42339 every 8 connection(s)
I20260928 09:02:27.483585 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:02:27.485826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16668
Sep 28 09:02:27 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:02:27.494100 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.486322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.0.98.62:36365)
0928 09:02:27.486440 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.486442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.486639 (+ 197us) server_negotiation.cc:407] Connection header received
0928 09:02:27.486665 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.486666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.486677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.486749 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:27.487448 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.488206 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.489222 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.489521 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.489576 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.491104 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:27.491106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:27.491106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:27.491114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:27.492878 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.493225 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.493226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.493226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.493299 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.493504 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.493504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.493505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.493652 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:27.493802 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.493877 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.493976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260928 09:02:27.494874 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.484324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:54095)
0928 09:02:27.484494 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.484530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.485074 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.485333 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.485337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.485931 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:27.486191 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.486196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.487260 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.487261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.488505 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.488507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.489078 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.489079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.489208 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.489871 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:27.489879 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:27.491454 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:27.493194 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:27.493200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:27.493202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:27.493514 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:27.493715 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:27.493715 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:27.493715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:27.493854 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:27.494120 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:27.494127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:27.494309 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.494426 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.494613 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260928 09:02:27.495177 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.484324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54095 (local address 127.0.98.62:36365)
0928 09:02:27.484397 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.484399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.484615 (+ 216us) server_negotiation.cc:407] Connection header received
0928 09:02:27.485166 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.485167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.485178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.485248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:27.486325 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.487163 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.488582 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.488961 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.489033 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.491568 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:27.491569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:27.491570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:27.491577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:27.493119 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.493584 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.493584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.493585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.493628 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.493920 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.493921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.493921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.494031 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:27.494091 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.495016 (+ 925us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.495094 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 09:02:27.495496 16797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36365
I20260928 09:02:27.495623 16797 heartbeater.cc:464] Registering TS with master...
I20260928 09:02:27.495828 16797 heartbeater.cc:511] Master 127.0.98.62:36365 requested a full tablet report, sending...
I20260928 09:02:27.496236 16339 ts_manager.cc:194] Registered new tserver with Master: 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339)
I20260928 09:02:27.496604 16339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54095
I20260928 09:02:27.505806 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:02:27.509192 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.506219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.98.62:36365)
0928 09:02:27.506315 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.506317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.506437 (+ 120us) server_negotiation.cc:407] Connection header received
0928 09:02:27.506545 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.506545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.506593 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.506674 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:02:27.506837 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.507342 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.508262 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.508456 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.508535 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.508745 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:02:27.508791 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 09:02:27.509085 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48682: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 09:02:27.509264 16530 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48682: token expired
W20260928 09:02:27.509281 16265 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36365)
I20260928 09:02:27.517014 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.509654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.0.98.62:36365)
0928 09:02:27.509694 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.509696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.509703 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:02:27.509724 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.509725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.509737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.509800 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:27.510296 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.510854 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.511679 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.512001 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.512081 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.513884 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:27.513886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:27.513886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:27.513895 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:27.515428 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.515905 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.515907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.515907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.515977 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:27.516230 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:27.516231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:27.516231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:27.516454 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:27.516592 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.516783 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.516884 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:02:27.518579 16339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:02:27.520565 16339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:02:27.523204 16450 tablet_service.cc:1467] Processing CreateTablet for tablet 0396955136834863a6c237e3240c9d36 (DEFAULT_TABLE table=test-table [id=09c929462d7d4095a0b2b9a5bb8bda69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:02:27.523415 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0396955136834863a6c237e3240c9d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:27.524956 16419 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap starting.
I20260928 09:02:27.525367 16419 tablet_bootstrap.cc:654] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:27.526008 16419 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: No bootstrap required, opened a new log
I20260928 09:02:27.526072 16419 ts_tablet_manager.cc:1397] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:27.526213 16419 raft_consensus.cc:348] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.526279 16419 raft_consensus.cc:374] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:27.526314 16419 raft_consensus.cc:729] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0f8e02746848d3813c0ad1df215e7b, State: Initialized, Role: FOLLOWER
I20260928 09:02:27.526384 16419 consensus_queue.cc:260] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.526502 16419 ts_tablet_manager.cc:1428] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:27.527311 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.522656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42339 (local address 127.0.0.1:53102)
0928 09:02:27.522708 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.522720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.522760 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.523225 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.523226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.523228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:27.523271 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.523272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.524606 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.524607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.525981 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.525985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.527027 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.527028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.527093 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.527097 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.527197 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.527243 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 09:02:27.527333 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.522859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.0.98.3:42339)
0928 09:02:27.523038 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.523043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.523060 (+ 17us) server_negotiation.cc:407] Connection header received
0928 09:02:27.523103 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.523105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.523122 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.523216 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:27.523378 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.524510 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.526098 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.526866 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.526933 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.526995 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.527216 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.527252 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260928 09:02:27.528230 16731 tablet_service.cc:1467] Processing CreateTablet for tablet 0396955136834863a6c237e3240c9d36 (DEFAULT_TABLE table=test-table [id=09c929462d7d4095a0b2b9a5bb8bda69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:02:27.528548 16731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0396955136834863a6c237e3240c9d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:27.529071 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.522656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35561 (local address 127.0.0.1:41872)
0928 09:02:27.525008 (+ 2352us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.525020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.525070 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.525287 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.525288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.525290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:27.525338 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.525339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.526507 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.526508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.527750 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.527753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.528778 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.528779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.528852 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.528857 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.528929 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.528971 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2331,"mutex_wait_us":60,"thread_start_us":100,"threads_started":1}
I20260928 09:02:27.529086 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.522845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.0.98.2:35561)
0928 09:02:27.523083 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.523089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.525131 (+ 2042us) server_negotiation.cc:407] Connection header received
0928 09:02:27.525189 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.525191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.525205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.525291 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:27.525386 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.526416 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.528034 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.528682 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.528775 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.528839 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.528943 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.528984 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":150,"threads_started":1}
I20260928 09:02:27.531440 16807 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Bootstrap starting.
I20260928 09:02:27.532004 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 0396955136834863a6c237e3240c9d36 (DEFAULT_TABLE table=test-table [id=09c929462d7d4095a0b2b9a5bb8bda69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:02:27.532150 16807 tablet_bootstrap.cc:654] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:27.532347 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0396955136834863a6c237e3240c9d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:02:27.532857 16807 log.cc:824] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:27.533686 16807 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: No bootstrap required, opened a new log
I20260928 09:02:27.533752 16807 ts_tablet_manager.cc:1397] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:02:27.535221 16809 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Bootstrap starting.
I20260928 09:02:27.535349 16807 raft_consensus.cc:348] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.535557 16807 raft_consensus.cc:374] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:27.535615 16807 raft_consensus.cc:729] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3691fbc148f64cff8b035867384c3849, State: Initialized, Role: FOLLOWER
I20260928 09:02:27.535808 16807 consensus_queue.cc:260] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.535874 16809 tablet_bootstrap.cc:654] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Neither blocks nor log segments found. Creating new log.
I20260928 09:02:27.536152 16807 ts_tablet_manager.cc:1428] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 09:02:27.536190 16797 heartbeater.cc:503] Master 127.0.98.62:36365 was elected leader, sending a full tablet report...
I20260928 09:02:27.536504 16809 log.cc:824] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:27.537408 16809 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: No bootstrap required, opened a new log
I20260928 09:02:27.537489 16809 ts_tablet_manager.cc:1397] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:02:27.538763 16809 raft_consensus.cc:348] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.538911 16809 raft_consensus.cc:374] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:02:27.538956 16809 raft_consensus.cc:729] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77a80d27e3a49789f85e0bce66bc924, State: Initialized, Role: FOLLOWER
I20260928 09:02:27.539060 16809 consensus_queue.cc:260] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.539318 16809 ts_tablet_manager.cc:1428] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 09:02:27.539378 16665 heartbeater.cc:503] Master 127.0.98.62:36365 was elected leader, sending a full tablet report...
W20260928 09:02:27.609393 16666 tablet.cc:2369] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:02:27.651396 16810 raft_consensus.cc:482] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:27.651515 16810 raft_consensus.cc:504] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.651868 16810 leader_election.cc:288] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561), 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271)
I20260928 09:02:27.655007 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.652091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35561 (local address 127.0.98.3:51485)
0928 09:02:27.652135 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.652146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.652191 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.652388 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.652389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.652391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:27.652470 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.652472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.653292 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.653293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.654085 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.654087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.654757 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.654758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.654838 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.654842 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.654894 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.654943 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 09:02:27.655047 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.652179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51485 (local address 127.0.98.2:35561)
0928 09:02:27.652232 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.652234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.652252 (+ 18us) server_negotiation.cc:407] Connection header received
0928 09:02:27.652278 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.652284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.652293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.652359 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:27.652550 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.653214 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.654198 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.654662 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.654730 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.654740 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.654909 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.654947 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 09:02:27.655169 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.652091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41271 (local address 127.0.98.3:58061)
0928 09:02:27.652467 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.652476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:27.652552 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:27.652717 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:27.652718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:27.652720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:02:27.652793 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.652795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.653434 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.653435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.654217 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:27.654220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.654899 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.654900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:27.654976 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.654981 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:02:27.655039 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:02:27.655082 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":158,"thread_start_us":132,"threads_started":1}
I20260928 09:02:27.655181 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.652166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58061 (local address 127.0.98.1:41271)
0928 09:02:27.652225 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.652228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.652559 (+ 331us) server_negotiation.cc:407] Connection header received
0928 09:02:27.652603 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.652604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.652614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.652711 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:02:27.652864 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.653355 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.654300 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.654804 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.654872 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.654892 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.655061 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.655112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 09:02:27.655498 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0396955136834863a6c237e3240c9d36" candidate_uuid: "3691fbc148f64cff8b035867384c3849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77a80d27e3a49789f85e0bce66bc924" is_pre_election: true
I20260928 09:02:27.655681 16619 raft_consensus.cc:2454] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3691fbc148f64cff8b035867384c3849 in term 0.
I20260928 09:02:27.655740 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0396955136834863a6c237e3240c9d36" candidate_uuid: "3691fbc148f64cff8b035867384c3849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0f8e02746848d3813c0ad1df215e7b" is_pre_election: true
I20260928 09:02:27.655917 16479 raft_consensus.cc:2454] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3691fbc148f64cff8b035867384c3849 in term 0.
I20260928 09:02:27.655958 16684 leader_election.cc:302] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924; no voters:
I20260928 09:02:27.656081 16810 raft_consensus.cc:2781] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:02:27.656148 16810 raft_consensus.cc:482] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:02:27.656196 16810 raft_consensus.cc:3037] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:02:27.656956 16810 raft_consensus.cc:504] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.657109 16810 leader_election.cc:288] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [CANDIDATE]: Term 1 election: Requested vote from peers d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561), 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271)
I20260928 09:02:27.657398 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0396955136834863a6c237e3240c9d36" candidate_uuid: "3691fbc148f64cff8b035867384c3849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77a80d27e3a49789f85e0bce66bc924"
I20260928 09:02:27.657471 16479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0396955136834863a6c237e3240c9d36" candidate_uuid: "3691fbc148f64cff8b035867384c3849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0f8e02746848d3813c0ad1df215e7b"
I20260928 09:02:27.657491 16619 raft_consensus.cc:3037] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:02:27.657546 16479 raft_consensus.cc:3037] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:02:27.658427 16619 raft_consensus.cc:2454] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3691fbc148f64cff8b035867384c3849 in term 1.
I20260928 09:02:27.658506 16479 raft_consensus.cc:2454] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3691fbc148f64cff8b035867384c3849 in term 1.
I20260928 09:02:27.658624 16684 leader_election.cc:302] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924; no voters:
I20260928 09:02:27.658711 16810 raft_consensus.cc:2781] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:02:27.658854 16810 raft_consensus.cc:686] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 1 LEADER]: Becoming Leader. State: Replica: 3691fbc148f64cff8b035867384c3849, State: Running, Role: LEADER
I20260928 09:02:27.658963 16810 consensus_queue.cc:237] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:27.659861 16338 catalog_manager.cc:5495] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3691fbc148f64cff8b035867384c3849 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3691fbc148f64cff8b035867384c3849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } health_report { overall_health: UNKNOWN } } }
I20260928 09:02:27.674225 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.671640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.0.98.3:42339)
0928 09:02:27.671696 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:27.671698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:27.671775 (+ 77us) server_negotiation.cc:407] Connection header received
0928 09:02:27.671879 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:27.671879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:27.671900 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:27.671978 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:02:27.672107 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.672580 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.673402 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:27.673601 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:27.673677 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:27.673850 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:02:27.673959 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:02:27.674055 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:27.674073 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 09:02:27.674097 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:02:27.676234 16619 raft_consensus.cc:1261] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Refusing update from remote peer 3691fbc148f64cff8b035867384c3849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:02:27.676249 16479 raft_consensus.cc:1261] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Refusing update from remote peer 3691fbc148f64cff8b035867384c3849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:02:27.676680 16814 consensus_queue.cc:1035] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:02:27.676864 16810 consensus_queue.cc:1035] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:02:27.679065 16817 mvcc.cc:205] Tried to move back new op lower bound from 7334240860879437824 to 7334240860812337152. Current Snapshot: MvccSnapshot[applied={T|T < 7334240860879437824}]
I20260928 09:02:27.680804 16818 mvcc.cc:205] Tried to move back new op lower bound from 7334240860879437824 to 7334240860812337152. Current Snapshot: MvccSnapshot[applied={T|T < 7334240860879437824 or (T in {7334240860879437824})}]
I20260928 09:02:27.681807 16338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:02:27.681905 16338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:02:27.682657 16338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Sending DeleteTablet for 3 replicas of tablet 0396955136834863a6c237e3240c9d36
I20260928 09:02:27.683012 16731 tablet_service.cc:1514] Processing DeleteTablet for tablet 0396955136834863a6c237e3240c9d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53102
I20260928 09:02:27.683058 16599 tablet_service.cc:1514] Processing DeleteTablet for tablet 0396955136834863a6c237e3240c9d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41872
I20260928 09:02:27.683072 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet 0396955136834863a6c237e3240c9d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:02:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59958
I20260928 09:02:27.683172 16536 tablet_replica.cc:330] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: stopping tablet replica
I20260928 09:02:27.683216 16536 raft_consensus.cc:2229] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:02:27.683276 16536 raft_consensus.cc:2258] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:02:27.683573 16536 ts_tablet_manager.cc:1905] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:02:27.683602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 16377
I20260928 09:02:27.683614 16828 tablet_replica.cc:330] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: stopping tablet replica
I20260928 09:02:27.685352 16828 raft_consensus.cc:2229] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:02:27.685516 16828 raft_consensus.cc:2258] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:02:27.685715 16827 tablet_replica.cc:330] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: stopping tablet replica
I20260928 09:02:27.685830 16827 raft_consensus.cc:2229] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:02:27.685936 16827 raft_consensus.cc:2258] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:02:27.686331 16827 ts_tablet_manager.cc:1905] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:02:27.688277 16827 ts_tablet_manager.cc:1918] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:02:27.688367 16827 log.cc:1196] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/0396955136834863a6c237e3240c9d36
I20260928 09:02:27.688443 16828 ts_tablet_manager.cc:1905] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:02:27.688741 16827 ts_tablet_manager.cc:1939] T 0396955136834863a6c237e3240c9d36 P 3691fbc148f64cff8b035867384c3849: Deleting consensus metadata
I20260928 09:02:27.689265 16326 catalog_manager.cc:4841] TS 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339): tablet 0396955136834863a6c237e3240c9d36 (table test-table [id=09c929462d7d4095a0b2b9a5bb8bda69]) successfully deleted
W20260928 09:02:27.689991 16324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41271 (error 108)
I20260928 09:02:27.690234 16828 ts_tablet_manager.cc:1918] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:02:27.690312 16828 log.cc:1196] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/0396955136834863a6c237e3240c9d36
I20260928 09:02:27.690574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 16531
I20260928 09:02:27.690596 16828 ts_tablet_manager.cc:1939] T 0396955136834863a6c237e3240c9d36 P d77a80d27e3a49789f85e0bce66bc924: Deleting consensus metadata
I20260928 09:02:27.690614 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.690490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41271 (local address 127.0.0.1:59968)
0928 09:02:27.690545 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.690582 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 09:02:27.691119 16324 catalog_manager.cc:4584] TS 6a0f8e02746848d3813c0ad1df215e7b (127.0.98.1:41271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0396955136834863a6c237e3240c9d36: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41271: connect: Connection refused (error 111)
I20260928 09:02:27.696861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 16668
I20260928 09:02:27.696990 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:27.696840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35561 (local address 127.0.0.1:41888)
0928 09:02:27.696915 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:27.696953 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 09:02:27.702407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 16306
I20260928 09:02:47.708729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--webserver_interface=127.0.98.62
--webserver_port=39451
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:47.791162 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:47.791406 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:47.791429 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:47.792537 16830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 09:02:47.792570 16830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 09:02:47.792583 16830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 09:02:47.792595 16830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:02:47.792608 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:47.792620 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:47.792703 16830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:02:47.792725 16830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:02:47.794270 16830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:47.794459 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:47.794621 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:47.799803 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:47.799899 16830 server_base.cc:1034] running on GCE node
W20260928 09:02:47.799839 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:47.799839 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:47.800267 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:47.800556 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:47.801721 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790586167801696 us; error 39 us; skew 500 ppm
Sep 28 09:02:47 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:47.809361 16830 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:02:47.809816 16830 webserver.cc:460] Webserver started at http://127.0.98.62:39451/ using document root <none> and password file <none>
I20260928 09:02:47.810009 16830 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:47.810050 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:47.811385 16830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:02:47.812084 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:47.812278 16830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:02:47.812386 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "c1767d174a174562888c7e2e54eadaba"
format_stamp: "Formatted at 2026-09-28 09:02:05 on dist-test-slave-286n"
server_key: "4fcc47bdb42c572f7961381281f398fd"
server_key_iv: "47c57f478227afc971d29c460812d9d4"
server_key_version: "encryptionkey@0"
I20260928 09:02:47.812834 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:47.825309 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:47.827273 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:47.827409 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:47.831458 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36365
I20260928 09:02:47.831518 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36365 every 8 connection(s)
I20260928 09:02:47.831903 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:02:47.833464 16898 sys_catalog.cc:263] Verifying existing consensus state
I20260928 09:02:47.834142 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap starting.
I20260928 09:02:47.834227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16830
I20260928 09:02:47.834542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41271
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36365
--webserver_port=35481
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:02:47.843343 16898 log.cc:824] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:47.845733 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:02:47.846061 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Bootstrap complete.
I20260928 09:02:47.848407 16898 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:47.848796 16898 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1767d174a174562888c7e2e54eadaba, State: Initialized, Role: FOLLOWER
I20260928 09:02:47.848987 16898 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:47.849064 16898 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:02:47.849115 16898 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:02:47.849164 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 2 FOLLOWER]: Advancing to term 3
I20260928 09:02:47.850170 16898 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:47.850291 16898 leader_election.cc:302] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1767d174a174562888c7e2e54eadaba; no voters:
I20260928 09:02:47.850497 16898 leader_election.cc:288] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 09:02:47.850548 16902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 3 FOLLOWER]: Leader election won for term 3
I20260928 09:02:47.850680 16902 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [term 3 LEADER]: Becoming Leader. State: Replica: c1767d174a174562888c7e2e54eadaba, State: Running, Role: LEADER
I20260928 09:02:47.850785 16898 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:02:47.850775 16902 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } }
I20260928 09:02:47.851078 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1767d174a174562888c7e2e54eadaba. Latest consensus state: current_term: 3 leader_uuid: "c1767d174a174562888c7e2e54eadaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260928 09:02:47.851157 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: This master's current role is: LEADER
I20260928 09:02:47.851084 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c1767d174a174562888c7e2e54eadaba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1767d174a174562888c7e2e54eadaba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36365 } } }
I20260928 09:02:47.851343 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba [sys.catalog]: This master's current role is: LEADER
I20260928 09:02:47.851372 16907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:02:47.852275 16907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:02:47.852583 16907 catalog_manager.cc:1259] Loaded cluster ID: 3181d671337b492c848601714f4f3237
I20260928 09:02:47.852623 16907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:02:47.853850 16907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:02:47.854032 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Loaded TSK: 0
I20260928 09:02:47.854075 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1767d174a174562888c7e2e54eadaba: Loaded TSK: 1
F20260928 09:02:47.854307 16907 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f81bb9bf9ad google::LogMessage::Fail() at ??:0
@ 0x7f81bb9c288a google::LogMessage::SendToLog() at ??:0
@ 0x7f81bb9bf4de google::LogMessage::Flush() at ??:0
@ 0x7f81bb9c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f81be12c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f81be129d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f81c39191cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f81c391948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f81c3919594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f81c38dfc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f81c38e9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f81bd88d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f81bd882954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f81be6e66db start_thread at ??:0
@ 0x7f81ba4bd61f clone at ??:0
W20260928 09:02:47.933487 16900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:47.933698 16900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:47.933725 16900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:47.934949 16900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:47.934993 16900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:47.935019 16900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:02:47.936487 16900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35481
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:47.936750 16900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:47.937029 16900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:47.943322 16942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:47.943346 16939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:47.943369 16940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:47.943672 16900 server_base.cc:1034] running on GCE node
I20260928 09:02:47.943823 16900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:47.944051 16900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:47.945199 16900 hybrid_clock.cc:648] HybridClock initialized: now 1790586167945165 us; error 53 us; skew 500 ppm
Sep 28 09:02:47 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:47.947831 16900 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:02:47.948338 16900 webserver.cc:460] Webserver started at http://127.0.98.1:35481/ using document root <none> and password file <none>
I20260928 09:02:47.948566 16900 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:47.948637 16900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:47.950179 16900 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:47.950989 16949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:47.951195 16900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:47.951323 16900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "6a0f8e02746848d3813c0ad1df215e7b"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "03e00b5434f12362e96f25fdf8b70726"
server_key_iv: "d72c9527838188e3ba99da189a83b3a0"
server_key_version: "encryptionkey@0"
I20260928 09:02:47.951683 16900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:47.960273 16900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:47.962543 16900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:47.962700 16900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:47.962954 16900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:47.963440 16956 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 09:02:47.965533 16900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 09:02:47.965615 16900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 09:02:47.965705 16900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 09:02:47.966418 16900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 09:02:47.966473 16900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 09:02:47.966560 16956 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap starting.
I20260928 09:02:47.968520 16956 log.cc:824] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:02:47.971980 16956 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 09:02:47.972327 16956 tablet_bootstrap.cc:492] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Bootstrap complete.
I20260928 09:02:47.972509 16956 ts_tablet_manager.cc:1397] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 09:02:47.973716 16900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41271
I20260928 09:02:47.973875 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41271 every 8 connection(s)
I20260928 09:02:47.974128 16900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:02:47.974789 16956 raft_consensus.cc:348] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:47.975152 16956 raft_consensus.cc:729] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0f8e02746848d3813c0ad1df215e7b, State: Initialized, Role: FOLLOWER
I20260928 09:02:47.975296 16956 consensus_queue.cc:260] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:47.975587 16956 ts_tablet_manager.cc:1428] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b: Time spent starting tablet: real 0.003s user 0.003s sys 0.004s
I20260928 09:02:47.980324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 16900
I20260928 09:02:47.980671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35561
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36365
--webserver_port=44199
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:48.073432 17067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:02:47.974943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:46443)
0928 09:02:47.975228 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:47.975277 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:47.975447 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:48.073202 (+ 97755us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
W20260928 09:02:48.073728 17065 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36365 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36365: Client connection negotiation failed: client connection to 127.0.98.62:36365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:02:48.074091 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.073970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:60441)
0928 09:02:48.074024 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.074062 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 09:02:48.074468 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.074382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:34877)
0928 09:02:48.074428 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.074452 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:02:48.074596 17065 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 09:02:48.080071 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:48.080273 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:48.080300 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:48.081545 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:48.081589 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:48.081615 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:02:48.083093 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44199
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:48.083364 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:48.083617 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:48.088972 17074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:48.088965 17075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:48.088944 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:48.089161 17069 server_base.cc:1034] running on GCE node
I20260928 09:02:48.089438 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:48.089715 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:48.090889 17069 hybrid_clock.cc:648] HybridClock initialized: now 1790586168090886 us; error 38 us; skew 500 ppm
Sep 28 09:02:48 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:48.093446 17069 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:02:48.093922 17069 webserver.cc:460] Webserver started at http://127.0.98.2:44199/ using document root <none> and password file <none>
I20260928 09:02:48.094161 17069 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:48.094238 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:48.095577 17069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:48.096205 17084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:48.096491 17069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:48.096594 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "d77a80d27e3a49789f85e0bce66bc924"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "cef1e500c053b2976976183cae88dc4c"
server_key_iv: "739b32ae55bfe1dc0df1f57a222bb652"
server_key_version: "encryptionkey@0"
I20260928 09:02:48.096990 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:48.116988 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:48.119500 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:48.119681 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:48.119943 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:48.120271 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:48.120306 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:48.120329 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:48.120376 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:48.126439 17069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35561
I20260928 09:02:48.126514 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35561 every 8 connection(s)
I20260928 09:02:48.126895 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:02:48.127988 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.127629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:48325)
0928 09:02:48.127778 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.127849 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
W20260928 09:02:48.128492 17198 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36365 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36365: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
I20260928 09:02:48.128836 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.128727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:55773)
0928 09:02:48.128780 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.128808 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 09:02:48.129089 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.129042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:50307)
0928 09:02:48.129061 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.129079 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 09:02:48.129231 17198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:02:48.136808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17069
I20260928 09:02:48.137076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42339
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36365
--webserver_port=36813
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:02:48.221539 17201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:02:48.221792 17201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:02:48.221835 17201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:02:48.223026 17201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:02:48.223100 17201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:02:48.223148 17201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:02:48.224726 17201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36813
--tserver_master_addrs=127.0.98.62:36365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:02:48.224993 17201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:02:48.225242 17201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:02:48.230439 17209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:48.230556 17201 server_base.cc:1034] running on GCE node
W20260928 09:02:48.230571 17207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:02:48.230671 17206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:02:48.230979 17201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:02:48.231231 17201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:02:48.232383 17201 hybrid_clock.cc:648] HybridClock initialized: now 1790586168232362 us; error 41 us; skew 500 ppm
Sep 28 09:02:48 dist-test-slave-286n krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:02:48.234984 17201 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:02:48.235507 17201 webserver.cc:460] Webserver started at http://127.0.98.3:36813/ using document root <none> and password file <none>
I20260928 09:02:48.235738 17201 fs_manager.cc:359] Metadata directory not provided
I20260928 09:02:48.235783 17201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:02:48.237231 17201 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:02:48.237802 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:48.237973 17201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:02:48.238070 17201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "3691fbc148f64cff8b035867384c3849"
format_stamp: "Formatted at 2026-09-28 09:02:06 on dist-test-slave-286n"
server_key: "17feb4ea2acde007c226fe2c8af3b6d9"
server_key_iv: "a9a6598983615120acf46b283957c570"
server_key_version: "encryptionkey@0"
I20260928 09:02:48.238428 17201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:02:48.257421 17201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:02:48.259925 17201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:02:48.260051 17201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:02:48.260239 17201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:02:48.260577 17201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:02:48.260639 17201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:48.260703 17201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:02:48.260725 17201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:02:48.266688 17201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42339
I20260928 09:02:48.266777 17329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42339 every 8 connection(s)
I20260928 09:02:48.267143 17201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:02:48.268463 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.267878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:48889)
0928 09:02:48.268025 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.268097 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260928 09:02:48.268772 17330 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36365 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36365: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
I20260928 09:02:48.269078 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.268982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:53323)
0928 09:02:48.269034 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.269060 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 09:02:48.269385 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:48.269307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:51713)
0928 09:02:48.269352 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:48.269373 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 09:02:48.269495 17330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 09:02:48.272045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17201
I20260928 09:02:49.075281 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.074994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:51399)
0928 09:02:49.075163 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:49.075212 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260928 09:02:49.129750 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.129527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:35811)
0928 09:02:49.129661 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:49.129695 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260928 09:02:49.205916 17336 raft_consensus.cc:482] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:49.206034 17336 raft_consensus.cc:504] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:49.206449 17336 leader_election.cc:288] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339), d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561)
Sep 28 09:02:49 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 09:02:49 dist-test-slave-286n krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 09:02:49.217681 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.206802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41029 (local address 127.0.98.3:42339)
0928 09:02:49.206947 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:49.206966 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:49.206995 (+ 29us) server_negotiation.cc:407] Connection header received
0928 09:02:49.207470 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:49.207472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:49.207525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:49.207855 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:49.208571 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.209677 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.210721 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.211148 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.211285 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:49.213875 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:49.213879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:49.213879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:49.213889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:49.215811 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216333 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:49.216336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:49.216337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:49.216456 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216729 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:49.216730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:49.216730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:49.217056 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:49.217116 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:49.217430 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 09:02:49.217561 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260928 09:02:49.217681 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.206619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35561 (local address 127.0.98.1:53117)
0928 09:02:49.206724 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:49.206735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:49.206796 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:49.207756 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:49.207759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:49.208564 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:49.208634 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:49.208637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.209886 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.209887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:49.210625 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:49.210628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.211228 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.211229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:49.211353 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:49.211962 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:49.211971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:49.214196 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:49.216146 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:49.216147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:49.216410 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:49.216733 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:49.216734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:49.216808 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:49.217184 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:49.217187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:49.217263 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 09:02:49.217377 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 09:02:49.217571 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":40}
I20260928 09:02:49.217700 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.206767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53117 (local address 127.0.98.2:35561)
0928 09:02:49.206911 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 09:02:49.206923 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:02:49.206959 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:02:49.207440 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:02:49.207443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:02:49.207488 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:02:49.207704 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:02:49.208869 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.209770 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.210721 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.211136 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.211245 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:49.214325 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:02:49.214329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:02:49.214330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:02:49.214341 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:02:49.216058 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216507 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:49.216510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:49.216511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:49.216621 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216874 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:02:49.216874 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:02:49.216875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:02:49.217104 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:02:49.217187 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 09:02:49.217431 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 09:02:49.217565 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":52,"threads_started":1}
I20260928 09:02:49.218175 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.206746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42339 (local address 127.0.98.1:41029)
0928 09:02:49.206939 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:49.206949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:02:49.206996 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:02:49.207806 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:02:49.207807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:02:49.208156 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:02:49.208427 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:49.208433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.209784 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.209785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:49.210627 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:02:49.210629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:02:49.211259 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:02:49.211260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:02:49.211354 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:02:49.212024 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:02:49.212028 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:02:49.213733 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:02:49.215909 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:49.215917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:49.215919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:49.216252 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:49.216526 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:02:49.216527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:02:49.216527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:02:49.216673 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:02:49.217128 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:02:49.217135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:02:49.217273 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 09:02:49.217377 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 09:02:49.217653 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":137,"thread_start_us":94,"threads_started":1}
W20260928 09:02:49.218557 16950 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
W20260928 09:02:49.218953 16953 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
I20260928 09:02:49.218995 16953 leader_election.cc:302] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0f8e02746848d3813c0ad1df215e7b; no voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924
I20260928 09:02:49.219107 17336 raft_consensus.cc:2726] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:02:49.270043 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:49.269835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:48945)
0928 09:02:49.269966 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:49.270008 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 09:02:50.076068 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:50.075844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:51127)
0928 09:02:50.075968 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:50.076013 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260928 09:02:50.130430 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:50.130221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:56033)
0928 09:02:50.130354 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:50.130390 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260928 09:02:50.270730 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:50.270522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:42545)
0928 09:02:50.270637 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:50.270677 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260928 09:02:51.020581 17344 raft_consensus.cc:482] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:51.020695 17344 raft_consensus.cc:504] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:51.020864 17344 leader_election.cc:288] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339), d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561)
W20260928 09:02:51.021273 16953 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
W20260928 09:02:51.021366 16950 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
I20260928 09:02:51.021471 16950 leader_election.cc:302] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0f8e02746848d3813c0ad1df215e7b; no voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924
I20260928 09:02:51.021567 17344 raft_consensus.cc:2726] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:02:51.076959 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:51.076626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:38369)
0928 09:02:51.076836 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:51.076886 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260928 09:02:51.131153 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:51.130929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:34145)
0928 09:02:51.131072 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:51.131113 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260928 09:02:51.271442 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:51.271232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:42891)
0928 09:02:51.271365 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:51.271399 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 09:02:52.077786 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:52.077556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:41663)
0928 09:02:52.077692 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:52.077739 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260928 09:02:52.131949 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:52.131714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:46095)
0928 09:02:52.131851 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:52.131900 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 09:02:52.272269 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:52.272005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:53545)
0928 09:02:52.272171 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:52.272222 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260928 09:02:53.078681 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:53.078372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:52903)
0928 09:02:53.078573 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:53.078619 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260928 09:02:53.132758 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:53.132500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:45499)
0928 09:02:53.132674 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:53.132715 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260928 09:02:53.273131 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:53.272907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:56697)
0928 09:02:53.273043 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:53.273086 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 09:02:54.079504 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:54.079272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:41709)
0928 09:02:54.079405 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:54.079455 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260928 09:02:54.133524 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:54.133292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:37079)
0928 09:02:54.133428 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:54.133475 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260928 09:02:54.176486 17356 raft_consensus.cc:482] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:54.176574 17356 raft_consensus.cc:504] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:54.176808 17356 leader_election.cc:288] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339), d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561)
W20260928 09:02:54.177121 16953 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
W20260928 09:02:54.177189 16950 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
I20260928 09:02:54.177222 16953 leader_election.cc:302] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0f8e02746848d3813c0ad1df215e7b; no voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924
I20260928 09:02:54.177316 17356 raft_consensus.cc:2726] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:02:54.273924 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:54.273702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:49449)
0928 09:02:54.273831 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:54.273876 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260928 09:02:55.080308 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:55.080070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:60095)
0928 09:02:55.080217 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:55.080260 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 09:02:55.134279 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:55.134049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:43957)
0928 09:02:55.134178 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:55.134223 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260928 09:02:55.274676 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:55.274434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:53115)
0928 09:02:55.274599 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:55.274629 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260928 09:02:56.081225 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:56.080968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:41041)
0928 09:02:56.081132 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:56.081178 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260928 09:02:56.135017 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:56.134780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:39281)
0928 09:02:56.134913 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:56.134969 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 09:02:56.275528 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:56.275219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:35843)
0928 09:02:56.275375 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:56.275480 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260928 09:02:57.082052 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:57.081798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:58881)
0928 09:02:57.081953 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:57.082001 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260928 09:02:57.135855 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:57.135604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:33557)
0928 09:02:57.135761 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:57.135810 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260928 09:02:57.276413 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:57.276123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:46289)
0928 09:02:57.276284 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:57.276339 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260928 09:02:58.083048 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:58.082658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:41993)
0928 09:02:58.082898 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:58.082962 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":176,"threads_started":1}
I20260928 09:02:58.136499 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:58.136301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:48273)
0928 09:02:58.136425 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:58.136454 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260928 09:02:58.277108 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:58.276895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:35199)
0928 09:02:58.277020 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:58.277058 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260928 09:02:59.021008 17370 raft_consensus.cc:482] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:02:59.021135 17370 raft_consensus.cc:504] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3691fbc148f64cff8b035867384c3849" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42339 } } peers { permanent_uuid: "d77a80d27e3a49789f85e0bce66bc924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35561 } } peers { permanent_uuid: "6a0f8e02746848d3813c0ad1df215e7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41271 } }
I20260928 09:02:59.021360 17370 leader_election.cc:288] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339), d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561)
W20260928 09:02:59.021795 16950 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d77a80d27e3a49789f85e0bce66bc924 (127.0.98.2:35561): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
W20260928 09:02:59.021874 16953 leader_election.cc:341] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3691fbc148f64cff8b035867384c3849 (127.0.98.3:42339): Not found: Tablet not found: 0396955136834863a6c237e3240c9d36
I20260928 09:02:59.021981 16953 leader_election.cc:302] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0f8e02746848d3813c0ad1df215e7b; no voters: 3691fbc148f64cff8b035867384c3849, d77a80d27e3a49789f85e0bce66bc924
I20260928 09:02:59.022145 17370 raft_consensus.cc:2726] T 0396955136834863a6c237e3240c9d36 P 6a0f8e02746848d3813c0ad1df215e7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 09:02:59.083847 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:59.083609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:55787)
0928 09:02:59.083757 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:59.083800 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260928 09:02:59.137182 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:59.136924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:51869)
0928 09:02:59.137074 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:59.137120 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 09:02:59.277792 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:02:59.277598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:45383)
0928 09:02:59.277713 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 09:02:59.277744 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 09:03:00.084795 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:00.084412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:49561)
0928 09:03:00.084627 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:00.084743 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":67,"threads_started":1}
I20260928 09:03:00.137897 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:00.137683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:39681)
0928 09:03:00.137806 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:00.137849 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 09:03:00.278530 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:00.278297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:57677)
0928 09:03:00.278434 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:00.278481 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 09:03:01.085530 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:01.085289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:57473)
0928 09:03:01.085433 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:01.085474 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260928 09:03:01.138684 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:01.138449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:55979)
0928 09:03:01.138593 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:01.138636 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260928 09:03:01.279251 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:01.279039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:47073)
0928 09:03:01.279161 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:01.279202 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 09:03:02.086234 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:02.085998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:48533)
0928 09:03:02.086146 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:02.086187 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260928 09:03:02.139431 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:02.139191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:47367)
0928 09:03:02.139343 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:02.139382 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260928 09:03:02.280035 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:02.279722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:35621)
0928 09:03:02.279925 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:02.279966 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":58,"threads_started":1}
I20260928 09:03:03.086915 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:03.086721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.1:41751)
0928 09:03:03.086836 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:03.086873 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260928 09:03:03.140235 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:03.140019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.2:36903)
0928 09:03:03.140147 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:03.140185 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260928 09:03:03.275503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 16900
I20260928 09:03:03.280594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17069
I20260928 09:03:03.281415 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:03.280730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36365 (local address 127.0.98.3:60903)
0928 09:03:03.281316 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:03.281355 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":63,"threads_started":1}
I20260928 09:03:03.285219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17201
2026-09-28T09:03:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58586 ms)
[----------] 30 tests from SecurityITest (263152 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:03 dist-test-slave-286n krb5kdc[17388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:03 dist-test-slave-286n krb5kdc[17388](info): set up 2 sockets
Sep 28 09:03:03 dist-test-slave-286n krb5kdc[17388](info): commencing operation
krb5kdc: starting...
W20260928 09:03:04.340894 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:04.460170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:04.541580 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:04.541798 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:04.541821 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:04.543037 17401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:04.543081 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:04.543095 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:04.543108 17401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:04.543162 17401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:04.544624 17401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:04.544934 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:04.545177 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:04.549923 17409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:04.549993 17406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:04.549999 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:04.550107 17401 server_base.cc:1034] running on GCE node
I20260928 09:03:04.550444 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:04.550694 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:04.551831 17401 hybrid_clock.cc:648] HybridClock initialized: now 1790586184551801 us; error 43 us; skew 500 ppm
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:04.559444 17401 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:04.559955 17401 webserver.cc:460] Webserver started at http://127.0.98.62:33277/ using document root <none> and password file <none>
I20260928 09:03:04.560204 17401 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:04.560251 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:04.560411 17401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:04.561556 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "f4c69583ff704a3e9756b491cb13515b"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "b15d314d845010cbd32699f3623a8481"
server_key_iv: "ace7909fd3509691e7e10a2ef02c4703"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.561934 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4c69583ff704a3e9756b491cb13515b"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "b15d314d845010cbd32699f3623a8481"
server_key_iv: "ace7909fd3509691e7e10a2ef02c4703"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.563251 17401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:04.563978 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.564157 17401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:04.564256 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "f4c69583ff704a3e9756b491cb13515b"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "b15d314d845010cbd32699f3623a8481"
server_key_iv: "ace7909fd3509691e7e10a2ef02c4703"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.564368 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:04.577708 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:04.579913 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:04.580103 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:04.584302 17401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45913
I20260928 09:03:04.584335 17468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45913 every 8 connection(s)
I20260928 09:03:04.584743 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:04.585783 17469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:04.588001 17469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b: Bootstrap starting.
I20260928 09:03:04.588531 17469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:04.588793 17469 log.cc:824] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:04.589469 17469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b: No bootstrap required, opened a new log
I20260928 09:03:04.590981 17469 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c69583ff704a3e9756b491cb13515b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260928 09:03:04.591114 17469 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:04.591137 17469 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c69583ff704a3e9756b491cb13515b, State: Initialized, Role: FOLLOWER
I20260928 09:03:04.591279 17469 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c69583ff704a3e9756b491cb13515b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260928 09:03:04.591364 17469 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:04.591394 17469 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:04.591425 17469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:04.591899 17469 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c69583ff704a3e9756b491cb13515b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260928 09:03:04.591984 17469 leader_election.cc:302] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c69583ff704a3e9756b491cb13515b; no voters:
I20260928 09:03:04.592100 17469 leader_election.cc:288] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:04.592149 17472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:04.592321 17472 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [term 1 LEADER]: Becoming Leader. State: Replica: f4c69583ff704a3e9756b491cb13515b, State: Running, Role: LEADER
I20260928 09:03:04.592360 17469 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:04.592443 17472 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c69583ff704a3e9756b491cb13515b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } }
I20260928 09:03:04.593050 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c69583ff704a3e9756b491cb13515b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c69583ff704a3e9756b491cb13515b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } } }
I20260928 09:03:04.593205 17474 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c69583ff704a3e9756b491cb13515b. Latest consensus state: current_term: 1 leader_uuid: "f4c69583ff704a3e9756b491cb13515b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c69583ff704a3e9756b491cb13515b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45913 } } }
I20260928 09:03:04.593273 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:04.593361 17474 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:04.593555 17482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:04.594023 17482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:04.594856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17401
I20260928 09:03:04.594983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:04.595211 392 external_mini_cluster.cc:1428] Setting key 9b771b67ae7a3ae1f90cb3d94810aeab
I20260928 09:03:04.595884 17482 catalog_manager.cc:1347] Generated new cluster ID: 1b04def81ad8445d93c6133b68077386
I20260928 09:03:04.595933 17482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:04.606837 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:04.595918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.98.62:45913)
0928 09:03:04.596066 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:04.596074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:04.596108 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:03:04.596225 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:04.596231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:04.596282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:04.597142 (+ 860us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:04.597249 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.598647 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.599707 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.600297 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.600715 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:04.603249 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:04.603253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:04.603254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:04.603266 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:04.605183 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:04.605590 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:04.605594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:04.605594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:04.605699 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:04.605929 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:04.605930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:04.605930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:04.606220 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:04.606352 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:04.606476 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 09:03:04.606607 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260928 09:03:04.607795 17482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:04.608489 17482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:04.614262 17482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b: Generated new TSK 0
I20260928 09:03:04.614439 17482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:03:04.648315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:04.749933 17495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:04.750133 17495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:04.750157 17495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:04.751336 17495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:04.751379 17495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:04.751432 17495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:03:04.753010 17495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:04.753250 17495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:04.753486 17495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:04.758584 17503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:04.758579 17501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:04.758646 17495 server_base.cc:1034] running on GCE node
W20260928 09:03:04.758668 17500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:04.759008 17495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:04.759212 17495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:04.760357 17495 hybrid_clock.cc:648] HybridClock initialized: now 1790586184760339 us; error 36 us; skew 500 ppm
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:04.762876 17495 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:03:04.763299 17495 webserver.cc:460] Webserver started at http://127.0.98.1:45823/ using document root <none> and password file <none>
I20260928 09:03:04.763513 17495 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:04.763558 17495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:04.763720 17495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:04.764904 17495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "64888c2d11fd4e0291c390fe9c7f1a90"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "c1cfa092dace81ff9be3fdec3eb6f192"
server_key_iv: "5f48408aab765233153edcdc58c15225"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.765259 17495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "64888c2d11fd4e0291c390fe9c7f1a90"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "c1cfa092dace81ff9be3fdec3eb6f192"
server_key_iv: "5f48408aab765233153edcdc58c15225"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.766515 17495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:04.767283 17510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.767493 17495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:04.767582 17495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "64888c2d11fd4e0291c390fe9c7f1a90"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "c1cfa092dace81ff9be3fdec3eb6f192"
server_key_iv: "5f48408aab765233153edcdc58c15225"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.767654 17495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:04.775869 17495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:04.777709 17495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:04.777865 17495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:04.778088 17495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:04.778389 17495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:04.778424 17495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.778466 17495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:04.778492 17495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.784245 17495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44713
I20260928 09:03:04.784334 17623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44713 every 8 connection(s)
I20260928 09:03:04.784682 17495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:04.793958 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:04.785421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40915 (local address 127.0.98.62:45913)
0928 09:03:04.785471 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:04.785473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:04.785656 (+ 183us) server_negotiation.cc:407] Connection header received
0928 09:03:04.786694 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:04.786695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:04.786709 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:04.786775 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:04.787385 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.787879 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.788939 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.789131 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.789185 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:04.791351 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:04.791352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:04.791353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:04.791358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:04.792581 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:04.793105 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:04.793106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:04.793107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:04.793157 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:04.793437 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:04.793437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:04.793437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:04.793541 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:04.793593 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:04.793827 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 09:03:04.793894 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:03:04.794212 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:04.785379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45913 (local address 127.0.98.1:40915)
0928 09:03:04.785537 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:04.785581 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:04.786595 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:04.786836 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:04.786840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:04.787098 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:04.787305 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:04.787309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.787954 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.787954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:04.788867 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:04.788870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.789200 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.789201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:04.789315 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:04.789882 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:04.789891 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:04.791252 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:04.792682 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:04.792688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:04.792689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:04.793026 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:04.793213 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:04.793214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:04.793214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:04.793335 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:04.793595 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:04.793600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:04.793715 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 09:03:04.793803 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:03:04.793922 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260928 09:03:04.794269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17495
I20260928 09:03:04.794386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:03:04.794612 392 external_mini_cluster.cc:1428] Setting key ebe58ab8f0e4abd5b1c9d7c6149cdbb8
I20260928 09:03:04.795035 17624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45913
I20260928 09:03:04.795132 17624 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:04.795341 17624 heartbeater.cc:511] Master 127.0.98.62:45913 requested a full tablet report, sending...
I20260928 09:03:04.795864 17433 ts_manager.cc:194] Registered new tserver with Master: 64888c2d11fd4e0291c390fe9c7f1a90 (127.0.98.1:44713)
I20260928 09:03:04.796463 17433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40915
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:03:04.819195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:04.903374 17629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:04.903666 17629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:04.903695 17629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:04.904881 17629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:04.904917 17629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:04.904942 17629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:03:04.906517 17629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:04.906708 17629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:04.906884 17629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:04.911834 17637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:04.911895 17634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:04.911845 17635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:04.912143 17629 server_base.cc:1034] running on GCE node
I20260928 09:03:04.912371 17629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:04.912565 17629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:04.913731 17629 hybrid_clock.cc:648] HybridClock initialized: now 1790586184913710 us; error 32 us; skew 500 ppm
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:04.916599 17629 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:03:04.917060 17629 webserver.cc:460] Webserver started at http://127.0.98.2:36219/ using document root <none> and password file <none>
I20260928 09:03:04.917292 17629 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:04.917357 17629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:04.917488 17629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:04.918517 17629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "229e57475aa34876867d2056907199b8"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "64be4876c3135d8b0da41ea53bde6dbd"
server_key_iv: "c762704a49e6dc94d85739249ee473cb"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.918880 17629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "229e57475aa34876867d2056907199b8"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "64be4876c3135d8b0da41ea53bde6dbd"
server_key_iv: "c762704a49e6dc94d85739249ee473cb"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.920171 17629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:04.920909 17644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.921073 17629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.921168 17629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "229e57475aa34876867d2056907199b8"
format_stamp: "Formatted at 2026-09-28 09:03:04 on dist-test-slave-286n"
server_key: "64be4876c3135d8b0da41ea53bde6dbd"
server_key_iv: "c762704a49e6dc94d85739249ee473cb"
server_key_version: "encryptionkey@0"
I20260928 09:03:04.921249 17629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:04.931794 17629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:04.934161 17629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:04.934325 17629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:04.934551 17629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:04.934860 17629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:04.934916 17629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.934957 17629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:04.934983 17629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:04.940809 17629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42369
I20260928 09:03:04.940851 17757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42369 every 8 connection(s)
I20260928 09:03:04.941162 17629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:03:04.944150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17629
I20260928 09:03:04.944301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:03:04.944506 392 external_mini_cluster.cc:1428] Setting key 4e94625ce93977a1278e348f11f44797
Sep 28 09:03:04 dist-test-slave-286n krb5kdc[17388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:04.951884 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:04.941885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47441 (local address 127.0.98.62:45913)
0928 09:03:04.941945 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:04.941947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:04.942162 (+ 215us) server_negotiation.cc:407] Connection header received
0928 09:03:04.943129 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:04.943131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:04.943144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:04.943191 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:04.943842 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.944462 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.945664 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.945866 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.945952 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:04.948744 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:04.948746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:04.948746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:04.948754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:04.950482 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:04.950886 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:04.950887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:04.950887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:04.950953 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:04.951186 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:04.951187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:04.951188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:04.951337 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:04.951416 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:04.951667 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 09:03:04.951769 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:03:04.951910 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:04.941852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45913 (local address 127.0.98.2:47441)
0928 09:03:04.942045 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:04.942081 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:04.943008 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:04.943299 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:04.943303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:04.943552 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:04.943745 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:04.943749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.944555 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.944556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:04.945575 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:04.945578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:04.945962 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:04.945963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:04.946090 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:04.946673 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:04.946685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:04.948614 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:04.950572 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:04.950577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:04.950579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:04.950805 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:04.951025 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:04.951026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:04.951026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:04.951122 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:04.951410 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:04.951414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:04.951534 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 09:03:04.951617 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:03:04.951748 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260928 09:03:04.952461 17758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45913
I20260928 09:03:04.952574 17758 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:04.952797 17758 heartbeater.cc:511] Master 127.0.98.62:45913 requested a full tablet report, sending...
I20260928 09:03:04.953238 17433 ts_manager.cc:194] Registered new tserver with Master: 229e57475aa34876867d2056907199b8 (127.0.98.2:42369)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 09:03:04.953747 17433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:03:04.971679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:05.052030 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:05.052284 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:05.052321 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:05.053467 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:05.053536 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:05.053580 17763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:03:05.055048 17763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:05.055320 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:05.055545 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:05.060473 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.060436 17771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.060488 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:05.060875 17763 server_base.cc:1034] running on GCE node
I20260928 09:03:05.061074 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:05.061259 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:05.062434 17763 hybrid_clock.cc:648] HybridClock initialized: now 1790586185062415 us; error 29 us; skew 500 ppm
Sep 28 09:03:05 dist-test-slave-286n krb5kdc[17388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:05.065052 17763 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:03:05.065471 17763 webserver.cc:460] Webserver started at http://127.0.98.3:44005/ using document root <none> and password file <none>
I20260928 09:03:05.065696 17763 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:05.065780 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:05.065925 17763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:05.067016 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ebb5ea3360a4407a44c282fc4dc7034"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "d3f769a6079610f87f63229c83497f73"
server_key_iv: "588931d087e766247a6ccc40023584bc"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.067432 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ebb5ea3360a4407a44c282fc4dc7034"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "d3f769a6079610f87f63229c83497f73"
server_key_iv: "588931d087e766247a6ccc40023584bc"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.068773 17763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:05.069505 17778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.069700 17763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:05.069805 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "4ebb5ea3360a4407a44c282fc4dc7034"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "d3f769a6079610f87f63229c83497f73"
server_key_iv: "588931d087e766247a6ccc40023584bc"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.069876 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:05.082988 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:05.085433 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:05.085603 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:05.085847 17763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:05.086161 17763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:05.086215 17763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.086256 17763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:05.086282 17763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.092133 17763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43489
I20260928 09:03:05.092238 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43489 every 8 connection(s)
I20260928 09:03:05.092614 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:03:05.096290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17763
I20260928 09:03:05.096388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:03:05.096679 392 external_mini_cluster.cc:1428] Setting key f9dd438c2dbc3ad2554908b6a9635559
Sep 28 09:03:05 dist-test-slave-286n krb5kdc[17388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:05.102056 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.093417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50723 (local address 127.0.98.62:45913)
0928 09:03:05.093480 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.093483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.093759 (+ 276us) server_negotiation.cc:407] Connection header received
0928 09:03:05.094712 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.094713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.094723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.094779 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:05.095366 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.095864 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.097094 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.097444 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.097516 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.099679 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:05.099680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:05.099681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:05.099687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:05.100978 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:05.101330 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:05.101331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:05.101331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:05.101379 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:05.101578 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:05.101578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:05.101579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:05.101678 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:05.101728 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.101931 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.101993 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:03:05.102296 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.093411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45913 (local address 127.0.98.3:50723)
0928 09:03:05.093594 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.093649 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.094604 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.094830 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.094834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.095095 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:05.095287 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.095291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.095944 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.095945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.097000 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.097002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.097546 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.097548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.097698 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.098271 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:05.098281 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:05.099588 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:05.101045 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:05.101051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:05.101053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:05.101266 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:05.101433 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:05.101434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:05.101434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:05.101526 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:05.101731 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:05.101735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:05.101843 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.101919 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.102030 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":113,"threads_started":1}
I20260928 09:03:05.102845 17892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45913
I20260928 09:03:05.102939 17892 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:05.103086 17892 heartbeater.cc:511] Master 127.0.98.62:45913 requested a full tablet report, sending...
I20260928 09:03:05.103457 17433 ts_manager.cc:194] Registered new tserver with Master: 4ebb5ea3360a4407a44c282fc4dc7034 (127.0.98.3:43489)
I20260928 09:03:05.103806 17433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50723
I20260928 09:03:05.108510 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:03:05.115846 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.109866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.0.98.62:45913)
0928 09:03:05.109942 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.109945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.110033 (+ 88us) server_negotiation.cc:407] Connection header received
0928 09:03:05.110150 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.110151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.110160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.110210 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:05.110766 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.111250 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.111995 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.112201 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.112283 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.113412 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:05.113414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:05.113414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:05.113421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:05.114705 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:05.115058 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:05.115058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:05.115059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:05.115108 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:05.115326 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:05.115326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:05.115327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:05.115453 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:05.115528 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.115704 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.115763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 09:03:05.117573 17433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:03:05.119143 17433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:03:05.125860 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.121563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43489 (local address 127.0.0.1:44610)
0928 09:03:05.122004 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.122012 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.122094 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.122427 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.122428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.122430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:05.122485 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.122487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.123774 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.123775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.124734 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.124737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.125552 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.125553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.125631 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.125646 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.125724 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.125775 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260928 09:03:05.125891 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.121674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.0.98.3:43489)
0928 09:03:05.121868 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.121873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.122067 (+ 194us) server_negotiation.cc:407] Connection header received
0928 09:03:05.122238 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.122241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.122260 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.122337 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:05.122554 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.123676 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.124846 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.125469 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.125547 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.125605 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.125748 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.125803 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260928 09:03:05.126314 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.121563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42369 (local address 127.0.0.1:59740)
0928 09:03:05.121935 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.121972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.122050 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.122371 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.122375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.122377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:05.122446 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.122449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.123806 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.123807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.125063 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.125066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.126058 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.126059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.126132 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.126137 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.126191 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.126239 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":25,"thread_start_us":54,"threads_started":1}
I20260928 09:03:05.126628 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.121688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.0.98.2:42369)
0928 09:03:05.121868 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.121873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.122139 (+ 266us) server_negotiation.cc:407] Connection header received
0928 09:03:05.122187 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.122190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.122205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.122271 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:05.122547 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.123698 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.125166 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.125986 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.126058 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.126114 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.126499 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.126541 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260928 09:03:05.126786 17826 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 (DEFAULT_TABLE table=test-table [id=8917860b83ab4be9b61f9265fc546e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:05.127169 17826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:05.127235 17692 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 (DEFAULT_TABLE table=test-table [id=8917860b83ab4be9b61f9265fc546e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:05.127566 17692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:05.127797 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.121751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44713 (local address 127.0.0.1:33746)
0928 09:03:05.122060 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.122071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.122153 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.122428 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.122429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.122430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:05.122484 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.122487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.125388 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.125389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.126670 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.126673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.127558 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.127559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.127630 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.127635 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.127697 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.127733 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 09:03:05.127847 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.121874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.0.98.1:44713)
0928 09:03:05.122048 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.122053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.122115 (+ 62us) server_negotiation.cc:407] Connection header received
0928 09:03:05.122248 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.122251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.122265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.122335 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:05.123810 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.125291 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.126768 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.127467 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.127546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.127600 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.127707 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.127745 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 09:03:05.129930 17909 tablet_bootstrap.cc:492] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Bootstrap starting.
I20260928 09:03:05.130647 17909 tablet_bootstrap.cc:654] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:05.130622 17910 tablet_bootstrap.cc:492] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Bootstrap starting.
I20260928 09:03:05.130780 17558 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 (DEFAULT_TABLE table=test-table [id=8917860b83ab4be9b61f9265fc546e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:05.131096 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:05.131259 17909 log.cc:824] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:05.131321 17910 tablet_bootstrap.cc:654] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:05.131867 17910 log.cc:824] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:05.132066 17909 tablet_bootstrap.cc:492] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: No bootstrap required, opened a new log
I20260928 09:03:05.132146 17909 ts_tablet_manager.cc:1397] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:05.132593 17910 tablet_bootstrap.cc:492] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: No bootstrap required, opened a new log
I20260928 09:03:05.132755 17910 ts_tablet_manager.cc:1397] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:05.134171 17913 tablet_bootstrap.cc:492] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: Bootstrap starting.
I20260928 09:03:05.134155 17909 raft_consensus.cc:348] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.134331 17909 raft_consensus.cc:374] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:05.134405 17909 raft_consensus.cc:729] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebb5ea3360a4407a44c282fc4dc7034, State: Initialized, Role: FOLLOWER
I20260928 09:03:05.134536 17909 consensus_queue.cc:260] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.134805 17909 ts_tablet_manager.cc:1428] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:03:05.134811 17913 tablet_bootstrap.cc:654] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:05.134788 17910 raft_consensus.cc:348] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.134935 17892 heartbeater.cc:503] Master 127.0.98.62:45913 was elected leader, sending a full tablet report...
I20260928 09:03:05.134980 17910 raft_consensus.cc:374] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:05.135027 17910 raft_consensus.cc:729] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e57475aa34876867d2056907199b8, State: Initialized, Role: FOLLOWER
I20260928 09:03:05.135135 17910 consensus_queue.cc:260] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.135366 17913 log.cc:824] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:05.135424 17758 heartbeater.cc:503] Master 127.0.98.62:45913 was elected leader, sending a full tablet report...
I20260928 09:03:05.135426 17910 ts_tablet_manager.cc:1428] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:03:05.136272 17913 tablet_bootstrap.cc:492] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: No bootstrap required, opened a new log
I20260928 09:03:05.136365 17913 ts_tablet_manager.cc:1397] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:05.137876 17913 raft_consensus.cc:348] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.138000 17913 raft_consensus.cc:374] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:05.138021 17913 raft_consensus.cc:729] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64888c2d11fd4e0291c390fe9c7f1a90, State: Initialized, Role: FOLLOWER
I20260928 09:03:05.138152 17913 consensus_queue.cc:260] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.138409 17913 ts_tablet_manager.cc:1428] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:05.138438 17624 heartbeater.cc:503] Master 127.0.98.62:45913 was elected leader, sending a full tablet report...
W20260928 09:03:05.191718 17759 tablet.cc:2369] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:03:05.285382 17625 tablet.cc:2369] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:03:05.343250 17893 tablet.cc:2369] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:05.466207 17914 raft_consensus.cc:482] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:03:05.466398 17914 raft_consensus.cc:504] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.466832 17914 leader_election.cc:288] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 229e57475aa34876867d2056907199b8 (127.0.98.2:42369), 64888c2d11fd4e0291c390fe9c7f1a90 (127.0.98.1:44713)
I20260928 09:03:05.469898 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.467084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57199 (local address 127.0.98.2:42369)
0928 09:03:05.467140 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.467143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.467196 (+ 53us) server_negotiation.cc:407] Connection header received
0928 09:03:05.467285 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.467286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.467306 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.467391 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:05.467543 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.468067 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.469170 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.469534 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.469617 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.469628 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.469784 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.469819 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:05.469921 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.467151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35663 (local address 127.0.98.1:44713)
0928 09:03:05.467212 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.467214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.467341 (+ 127us) server_negotiation.cc:407] Connection header received
0928 09:03:05.467487 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.467488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.467498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.467569 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:05.467728 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.468180 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.469213 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.469572 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.469652 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.469661 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.469812 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.469847 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:03:05.469945 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.467023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42369 (local address 127.0.98.3:57199)
0928 09:03:05.467145 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.467157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.467215 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.467374 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.467375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.467376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:05.467452 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.467454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.468162 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.468163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.469071 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.469073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.469616 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.469617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.469699 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.469702 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.469776 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.469815 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":26}
I20260928 09:03:05.469996 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.467071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44713 (local address 127.0.98.3:35663)
0928 09:03:05.467260 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.467278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.467357 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.467572 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.467573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.467574 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:05.467647 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.467648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.468267 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.468268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.469115 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.469117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.469653 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.469654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.469737 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.469739 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.469807 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.469845 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":113,"threads_started":1}
I20260928 09:03:05.470436 17712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13f046eb5b4ac2a8f77b8b980c5aa7" candidate_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229e57475aa34876867d2056907199b8" is_pre_election: true
I20260928 09:03:05.470511 17578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13f046eb5b4ac2a8f77b8b980c5aa7" candidate_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" is_pre_election: true
I20260928 09:03:05.470615 17712 raft_consensus.cc:2454] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ebb5ea3360a4407a44c282fc4dc7034 in term 0.
I20260928 09:03:05.470671 17578 raft_consensus.cc:2454] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ebb5ea3360a4407a44c282fc4dc7034 in term 0.
I20260928 09:03:05.470854 17782 leader_election.cc:302] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229e57475aa34876867d2056907199b8, 4ebb5ea3360a4407a44c282fc4dc7034; no voters:
I20260928 09:03:05.471017 17914 raft_consensus.cc:2781] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:03:05.471082 17914 raft_consensus.cc:482] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:03:05.471102 17914 raft_consensus.cc:3037] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:05.471792 17914 raft_consensus.cc:504] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.471936 17914 leader_election.cc:288] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [CANDIDATE]: Term 1 election: Requested vote from peers 229e57475aa34876867d2056907199b8 (127.0.98.2:42369), 64888c2d11fd4e0291c390fe9c7f1a90 (127.0.98.1:44713)
I20260928 09:03:05.472107 17712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13f046eb5b4ac2a8f77b8b980c5aa7" candidate_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229e57475aa34876867d2056907199b8"
I20260928 09:03:05.472138 17578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13f046eb5b4ac2a8f77b8b980c5aa7" candidate_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64888c2d11fd4e0291c390fe9c7f1a90"
I20260928 09:03:05.472198 17578 raft_consensus.cc:3037] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:05.472209 17712 raft_consensus.cc:3037] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:05.472949 17578 raft_consensus.cc:2454] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ebb5ea3360a4407a44c282fc4dc7034 in term 1.
I20260928 09:03:05.473014 17712 raft_consensus.cc:2454] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ebb5ea3360a4407a44c282fc4dc7034 in term 1.
I20260928 09:03:05.473122 17781 leader_election.cc:302] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ebb5ea3360a4407a44c282fc4dc7034, 64888c2d11fd4e0291c390fe9c7f1a90; no voters:
I20260928 09:03:05.473232 17914 raft_consensus.cc:2781] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:05.473384 17914 raft_consensus.cc:686] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 1 LEADER]: Becoming Leader. State: Replica: 4ebb5ea3360a4407a44c282fc4dc7034, State: Running, Role: LEADER
I20260928 09:03:05.473489 17914 consensus_queue.cc:237] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } }
I20260928 09:03:05.474336 17432 catalog_manager.cc:5495] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ebb5ea3360a4407a44c282fc4dc7034 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ebb5ea3360a4407a44c282fc4dc7034" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 } health_report { overall_health: UNKNOWN } } }
I20260928 09:03:05.480624 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.478445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.0.98.3:43489)
0928 09:03:05.478488 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.478490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.478513 (+ 23us) server_negotiation.cc:407] Connection header received
0928 09:03:05.478574 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.478575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.478590 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.478649 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:05.478727 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.479174 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.479921 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.480113 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.480165 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.480283 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:05.480374 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:05.480432 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.480486 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.480500 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:03:05.482385 17712 raft_consensus.cc:1261] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 1 FOLLOWER]: Refusing update from remote peer 4ebb5ea3360a4407a44c282fc4dc7034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:05.482386 17578 raft_consensus.cc:1261] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 1 FOLLOWER]: Refusing update from remote peer 4ebb5ea3360a4407a44c282fc4dc7034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:05.482728 17914 consensus_queue.cc:1035] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64888c2d11fd4e0291c390fe9c7f1a90" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:05.482856 17919 consensus_queue.cc:1035] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229e57475aa34876867d2056907199b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:05.485246 17923 mvcc.cc:205] Tried to move back new op lower bound from 7334241015733841920 to 7334241015700856832. Current Snapshot: MvccSnapshot[applied={T|T < 7334241015733841920}]
I20260928 09:03:05.485908 17924 mvcc.cc:205] Tried to move back new op lower bound from 7334241015733841920 to 7334241015700856832. Current Snapshot: MvccSnapshot[applied={T|T < 7334241015733841920}]
I20260928 09:03:05.487295 17433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:05.487437 17433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:05.488298 17433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4c69583ff704a3e9756b491cb13515b: Sending DeleteTablet for 3 replicas of tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7
I20260928 09:03:05.488734 17692 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59740
I20260928 09:03:05.488771 17826 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44610
I20260928 09:03:05.488941 17558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33746
I20260928 09:03:05.489035 17933 tablet_replica.cc:330] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: stopping tablet replica
I20260928 09:03:05.489145 17933 raft_consensus.cc:2229] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:03:05.489156 17932 tablet_replica.cc:330] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: stopping tablet replica
I20260928 09:03:05.489240 17932 raft_consensus.cc:2229] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:05.489259 17933 raft_consensus.cc:2258] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:05.489364 17934 tablet_replica.cc:330] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90: stopping tablet replica
I20260928 09:03:05.489462 17934 raft_consensus.cc:2229] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:05.489531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17495
I20260928 09:03:05.489619 17932 raft_consensus.cc:2258] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:05.489686 17933 ts_tablet_manager.cc:1905] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:05.489784 17934 raft_consensus.cc:2258] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 64888c2d11fd4e0291c390fe9c7f1a90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:05.491708 17933 ts_tablet_manager.cc:1918] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:05.491775 17933 log.cc:1196] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/9e13f046eb5b4ac2a8f77b8b980c5aa7
I20260928 09:03:05.492094 17933 ts_tablet_manager.cc:1939] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 4ebb5ea3360a4407a44c282fc4dc7034: Deleting consensus metadata
I20260928 09:03:05.492496 17419 catalog_manager.cc:4841] TS 4ebb5ea3360a4407a44c282fc4dc7034 (127.0.98.3:43489): tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7 (table test-table [id=8917860b83ab4be9b61f9265fc546e01]) successfully deleted
I20260928 09:03:05.497001 17932 ts_tablet_manager.cc:1905] T 9e13f046eb5b4ac2a8f77b8b980c5aa7 P 229e57475aa34876867d2056907199b8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 09:03:05.497376 17419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44713 (error 108)
I20260928 09:03:05.497592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17629
I20260928 09:03:05.497920 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.497751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44713 (local address 127.0.0.1:33752)
0928 09:03:05.497828 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.497878 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 09:03:05.498310 17419 catalog_manager.cc:4584] TS 64888c2d11fd4e0291c390fe9c7f1a90 (127.0.98.1:44713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e13f046eb5b4ac2a8f77b8b980c5aa7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44713: connect: Connection refused (error 111)
I20260928 09:03:05.502462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17763
I20260928 09:03:05.502518 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.502397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42369 (local address 127.0.0.1:59746)
0928 09:03:05.502446 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.502474 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 09:03:05.507135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17401
2026-09-28T09:03:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2222 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T09:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:05Z Disabled control of system clock
I20260928 09:03:05.535707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44729
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:05.618625 17939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:05.618881 17939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:05.618906 17939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:05.620273 17939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:05.620326 17939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:05.620342 17939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:05.620384 17939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:05.620419 17939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:05.621912 17939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:05.622195 17939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:05.622527 17939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:05.627524 17944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:05.627573 17939 server_base.cc:1034] running on GCE node
W20260928 09:03:05.627530 17947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.627543 17945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:05.628006 17939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:05.628248 17939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:05.629397 17939 hybrid_clock.cc:648] HybridClock initialized: now 1790586185629388 us; error 49 us; skew 500 ppm
I20260928 09:03:05.630630 17939 webserver.cc:460] Webserver started at http://127.0.98.62:46607/ using document root <none> and password file <none>
I20260928 09:03:05.630898 17939 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:05.630947 17939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:05.631090 17939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:05.632157 17939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "4aaf3932de684c008ad84c9499bb2df6"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "323ac6bcaa1e2cbb4fd37f4b9d8dca50"
server_key_iv: "23388689f571c42487a2d579a0e6041a"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.632534 17939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "4aaf3932de684c008ad84c9499bb2df6"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "323ac6bcaa1e2cbb4fd37f4b9d8dca50"
server_key_iv: "23388689f571c42487a2d579a0e6041a"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.634117 17939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:05.635003 17953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.635190 17939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:05.635286 17939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "4aaf3932de684c008ad84c9499bb2df6"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "323ac6bcaa1e2cbb4fd37f4b9d8dca50"
server_key_iv: "23388689f571c42487a2d579a0e6041a"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.635392 17939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:05.646749 17939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:05.647080 17939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:05.647270 17939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:05.651489 17939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44729
I20260928 09:03:05.651522 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44729 every 8 connection(s)
I20260928 09:03:05.651882 17939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:05.652987 18006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:05.655252 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6: Bootstrap starting.
I20260928 09:03:05.655790 18006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:05.656034 18006 log.cc:824] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:05.656706 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6: No bootstrap required, opened a new log
I20260928 09:03:05.658234 18006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaf3932de684c008ad84c9499bb2df6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44729 } }
I20260928 09:03:05.658391 18006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:05.658416 18006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aaf3932de684c008ad84c9499bb2df6, State: Initialized, Role: FOLLOWER
I20260928 09:03:05.658562 18006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaf3932de684c008ad84c9499bb2df6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44729 } }
I20260928 09:03:05.658643 18006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:05.658664 18006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:05.658692 18006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:05.659157 18006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaf3932de684c008ad84c9499bb2df6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44729 } }
I20260928 09:03:05.659238 18006 leader_election.cc:302] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aaf3932de684c008ad84c9499bb2df6; no voters:
I20260928 09:03:05.659366 18006 leader_election.cc:288] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:05.659430 18009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:05.659601 18009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [term 1 LEADER]: Becoming Leader. State: Replica: 4aaf3932de684c008ad84c9499bb2df6, State: Running, Role: LEADER
I20260928 09:03:05.659657 18006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:05.659737 18009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaf3932de684c008ad84c9499bb2df6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44729 } }
I20260928 09:03:05.660280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 17939
I20260928 09:03:05.660336 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aaf3932de684c008ad84c9499bb2df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaf3932de684c008ad84c9499bb2df6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44729 } } }
I20260928 09:03:05.660409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:05.660413 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:05.660614 392 external_mini_cluster.cc:1428] Setting key 1810ec968034069165f95561b7a7e07a
I20260928 09:03:05.660840 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aaf3932de684c008ad84c9499bb2df6. Latest consensus state: current_term: 1 leader_uuid: "4aaf3932de684c008ad84c9499bb2df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaf3932de684c008ad84c9499bb2df6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44729 } } }
I20260928 09:03:05.660918 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:05.661032 18018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:05.661897 18018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:05.663584 18018 catalog_manager.cc:1347] Generated new cluster ID: 0b0a031d0ea44d1f807323f56fee1648
I20260928 09:03:05.663637 18018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:05.666612 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.661294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.0.98.62:44729)
0928 09:03:05.661490 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.661499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.661530 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:03:05.661648 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.661657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.661730 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.661876 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:05.662125 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.663325 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.664431 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.665062 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.665184 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.665914 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:05.665918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:05.665918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:05.666132 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:05.666170 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:05.666184 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:05.666244 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.666342 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.666413 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260928 09:03:05.670826 18018 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:05.671329 18018 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:05.675276 18018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6: Generated new TSK 0
I20260928 09:03:05.675442 18018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:05.676919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44729
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:05.759611 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:05.759805 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:05.759828 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:05.761147 18030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:05.761186 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:05.761212 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:03:05.762562 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:05.762799 18030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:05.763098 18030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:05.768123 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.768085 18038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.768076 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:05.768267 18030 server_base.cc:1034] running on GCE node
I20260928 09:03:05.768538 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:05.768843 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:05.770028 18030 hybrid_clock.cc:648] HybridClock initialized: now 1790586185770012 us; error 25 us; skew 500 ppm
I20260928 09:03:05.771474 18030 webserver.cc:460] Webserver started at http://127.0.98.1:45345/ using document root <none> and password file <none>
I20260928 09:03:05.771715 18030 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:05.771771 18030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:05.771888 18030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:05.773242 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "71980253e0d34184941add43d1c41980"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "d30e091afcb43beeb0bc96502fbab7f2"
server_key_iv: "45fa32390274f4fd6ab91ea7f9a3c980"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.773644 18030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71980253e0d34184941add43d1c41980"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "d30e091afcb43beeb0bc96502fbab7f2"
server_key_iv: "45fa32390274f4fd6ab91ea7f9a3c980"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.775183 18030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:05.776018 18044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.776226 18030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:05.776343 18030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "71980253e0d34184941add43d1c41980"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "d30e091afcb43beeb0bc96502fbab7f2"
server_key_iv: "45fa32390274f4fd6ab91ea7f9a3c980"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.776419 18030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:05.789350 18030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:05.789662 18030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:05.789783 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:05.790007 18030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:05.790354 18030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:05.790387 18030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.790410 18030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:05.790452 18030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.796003 18030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36509
I20260928 09:03:05.796046 18157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36509 every 8 connection(s)
I20260928 09:03:05.796408 18030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:03:05.801579 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.797191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56237 (local address 127.0.98.62:44729)
0928 09:03:05.797251 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.797253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.797478 (+ 225us) server_negotiation.cc:407] Connection header received
0928 09:03:05.797991 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.797992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.798005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.798069 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:05.798727 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.799213 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.800214 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.800399 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.800455 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.801355 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:05.801356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:05.801357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:05.801361 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:05.801372 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:05.801376 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:05.801416 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.801491 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.801517 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:03:05.801815 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.797147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44729 (local address 127.0.98.1:56237)
0928 09:03:05.797343 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.797385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.797870 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.798147 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.798150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.798158 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:05.798647 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.798651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.799295 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.799295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.800123 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.800126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.800477 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.800478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.800586 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.801222 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:05.801223 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:05.801252 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:05.801275 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:05.801424 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:05.801426 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:05.801428 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.801475 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.801554 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 09:03:05.802060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18030
I20260928 09:03:05.802193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:03:05.802386 392 external_mini_cluster.cc:1428] Setting key f9242330d69e11c49a96bc7a05909dd8
I20260928 09:03:05.802408 18158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44729
I20260928 09:03:05.802510 18158 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:05.802713 18158 heartbeater.cc:511] Master 127.0.98.62:44729 requested a full tablet report, sending...
I20260928 09:03:05.803320 17970 ts_manager.cc:194] Registered new tserver with Master: 71980253e0d34184941add43d1c41980 (127.0.98.1:36509)
I20260928 09:03:05.803428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44729
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 09:03:05.804232 17970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56237
W20260928 09:03:05.883783 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:05.884024 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:05.884058 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:05.885221 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:05.885284 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:05.885325 18161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:03:05.886664 18161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:05.886936 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:05.887228 18161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:05.892271 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.892271 18166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:05.892328 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:05.892537 18161 server_base.cc:1034] running on GCE node
I20260928 09:03:05.892853 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:05.893093 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:05.894246 18161 hybrid_clock.cc:648] HybridClock initialized: now 1790586185894223 us; error 28 us; skew 500 ppm
I20260928 09:03:05.895514 18161 webserver.cc:460] Webserver started at http://127.0.98.2:44997/ using document root <none> and password file <none>
I20260928 09:03:05.895762 18161 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:05.895809 18161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:05.895979 18161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:05.897102 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "f15e84a5f1974dcab38b7f352c0299ca"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "e747119e396ea4e89a923a5c4ad1f55e"
server_key_iv: "9cc532a63dde8f2bb4adbc50ae1341c1"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.897459 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f15e84a5f1974dcab38b7f352c0299ca"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "e747119e396ea4e89a923a5c4ad1f55e"
server_key_iv: "9cc532a63dde8f2bb4adbc50ae1341c1"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.898795 18161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:05.899479 18175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.899642 18161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:05.899739 18161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "f15e84a5f1974dcab38b7f352c0299ca"
format_stamp: "Formatted at 2026-09-28 09:03:05 on dist-test-slave-286n"
server_key: "e747119e396ea4e89a923a5c4ad1f55e"
server_key_iv: "9cc532a63dde8f2bb4adbc50ae1341c1"
server_key_version: "encryptionkey@0"
I20260928 09:03:05.899818 18161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:05.912747 18161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:05.913038 18161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:05.913183 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:05.913412 18161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:05.913740 18161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:05.913801 18161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.913842 18161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:05.913868 18161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:05.919543 18161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41185
I20260928 09:03:05.919620 18288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41185 every 8 connection(s)
I20260928 09:03:05.919941 18161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:03:05.925215 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.920709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54645 (local address 127.0.98.62:44729)
0928 09:03:05.920777 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:05.920779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:05.920981 (+ 202us) server_negotiation.cc:407] Connection header received
0928 09:03:05.921524 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:05.921526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:05.921538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:05.921601 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:05.922281 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.922798 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.923777 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.923992 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.924078 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.924926 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:05.924928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:05.924928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:05.924934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:05.924949 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:05.924953 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:05.925007 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:05.925100 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 09:03:05.925131 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:03:05.925446 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:05.920622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44729 (local address 127.0.98.2:54645)
0928 09:03:05.920857 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:05.920890 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:05.921408 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:05.921671 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:05.921675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:05.921685 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:05.922176 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.922180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.922894 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.922895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.923677 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:05.923679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:05.924088 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:05.924089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:05.924227 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:05.924789 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:05.924789 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:05.924816 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:05.924836 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:05.925010 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:05.925015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:05.925017 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:03:05.925102 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 09:03:05.925197 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260928 09:03:05.925951 18289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44729
I20260928 09:03:05.926046 18289 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:05.926187 18289 heartbeater.cc:511] Master 127.0.98.62:44729 requested a full tablet report, sending...
I20260928 09:03:05.926501 17970 ts_manager.cc:194] Registered new tserver with Master: f15e84a5f1974dcab38b7f352c0299ca (127.0.98.2:41185)
I20260928 09:03:05.926838 17970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54645
I20260928 09:03:05.928378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18161
I20260928 09:03:05.928485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:03:05.928715 392 external_mini_cluster.cc:1428] Setting key cd6d3bb413448ec2b0b8107660fbdf74
I20260928 09:03:05.929627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44729
--builtin_ntp_servers=127.0.98.20:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:06.011242 18292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:06.011485 18292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:06.011533 18292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:06.012703 18292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:06.012768 18292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:06.012809 18292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:03:06.014179 18292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:06.014451 18292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:06.014762 18292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:06.019709 18300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:06.019711 18298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:06.019711 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:06.019907 18292 server_base.cc:1034] running on GCE node
I20260928 09:03:06.020164 18292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:06.020426 18292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:06.021584 18292 hybrid_clock.cc:648] HybridClock initialized: now 1790586186021554 us; error 45 us; skew 500 ppm
I20260928 09:03:06.022780 18292 webserver.cc:460] Webserver started at http://127.0.98.3:40527/ using document root <none> and password file <none>
I20260928 09:03:06.022987 18292 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:06.023027 18292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:06.023116 18292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:06.024292 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "e6117730fab14d62bcc827dd48c96bb5"
format_stamp: "Formatted at 2026-09-28 09:03:06 on dist-test-slave-286n"
server_key: "bed5dd6554c20c85c781dfc0e3a4f6b2"
server_key_iv: "946486cc9fda53b142f1e8a416aa7e37"
server_key_version: "encryptionkey@0"
I20260928 09:03:06.024585 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e6117730fab14d62bcc827dd48c96bb5"
format_stamp: "Formatted at 2026-09-28 09:03:06 on dist-test-slave-286n"
server_key: "bed5dd6554c20c85c781dfc0e3a4f6b2"
server_key_iv: "946486cc9fda53b142f1e8a416aa7e37"
server_key_version: "encryptionkey@0"
I20260928 09:03:06.025935 18292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:06.026623 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:06.026839 18292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:06.026939 18292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "e6117730fab14d62bcc827dd48c96bb5"
format_stamp: "Formatted at 2026-09-28 09:03:06 on dist-test-slave-286n"
server_key: "bed5dd6554c20c85c781dfc0e3a4f6b2"
server_key_iv: "946486cc9fda53b142f1e8a416aa7e37"
server_key_version: "encryptionkey@0"
I20260928 09:03:06.027012 18292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:06.033703 18292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:06.034025 18292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:06.034178 18292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:06.034385 18292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:06.034696 18292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:06.034749 18292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:06.034789 18292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:06.034817 18292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:06.040794 18292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45881
I20260928 09:03:06.040848 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45881 every 8 connection(s)
I20260928 09:03:06.041186 18292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:03:06.044194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18292
I20260928 09:03:06.044293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:03:06.044510 392 external_mini_cluster.cc:1428] Setting key 94fff74f7ee826afedabf5eac98edc98
I20260928 09:03:06.046895 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.041985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58009 (local address 127.0.98.62:44729)
0928 09:03:06.042050 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.042053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.042281 (+ 228us) server_negotiation.cc:407] Connection header received
0928 09:03:06.042825 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.042826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.042840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.042891 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:06.043563 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.044107 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.045557 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.045775 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.045861 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.046624 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:06.046626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:06.046626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:06.046631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:06.046645 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:06.046649 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:06.046705 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.046785 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.046816 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:03:06.047137 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.041954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44729 (local address 127.0.98.3:58009)
0928 09:03:06.042136 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.042192 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:06.042710 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:06.042957 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:06.042961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:06.042971 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:06.043466 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.043470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.044206 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.044208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.045478 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.045481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.045869 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.045870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.046011 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.046485 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:06.046485 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:06.046513 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:06.046534 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:06.046712 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:06.046717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:06.046721 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:06.046787 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 09:03:06.046885 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260928 09:03:06.047598 18420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44729
I20260928 09:03:06.047724 18420 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:06.047904 18420 heartbeater.cc:511] Master 127.0.98.62:44729 requested a full tablet report, sending...
I20260928 09:03:06.048317 17970 ts_manager.cc:194] Registered new tserver with Master: e6117730fab14d62bcc827dd48c96bb5 (127.0.98.3:45881)
I20260928 09:03:06.048768 17970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58009
I20260928 09:03:06.056330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:03:06.060906 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.057775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.0.98.62:44729)
0928 09:03:06.057844 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.057845 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.057992 (+ 147us) server_negotiation.cc:407] Connection header received
0928 09:03:06.058060 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.058061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.058086 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.058140 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:06.058324 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.058819 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.059685 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.059948 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.060038 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.060651 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:06.060653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:06.060653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:06.060659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:06.060672 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:06.060676 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:06.060730 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.060788 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.060813 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:03:06.062448 17970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:03:06.063973 17970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:03:06.070318 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.066327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.0.98.2:41185)
0928 09:03:06.066510 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.066513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.066524 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:03:06.066567 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.066568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.066584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.066678 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:06.066897 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.067739 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.069428 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.069910 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.070002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.070097 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.070189 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.070227 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260928 09:03:06.070307 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.066093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41185 (local address 127.0.0.1:47362)
0928 09:03:06.066476 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.066484 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:06.066523 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:06.066711 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:06.066714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:06.066718 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:06.066805 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.066807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.067829 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.067830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.069350 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.069352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.070000 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.070001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.070075 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.070088 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:03:06.070174 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 09:03:06.070228 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":10,"thread_start_us":101,"threads_started":1}
I20260928 09:03:06.070430 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.066071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36509 (local address 127.0.0.1:45296)
0928 09:03:06.066258 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.066302 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:06.066374 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:06.067063 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:06.067064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:06.067066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:06.067111 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.067113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.068298 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.068299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.069102 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.069105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.070136 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.070136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.070236 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.070243 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 09:03:06.070294 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:03:06.070348 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":45,"thread_start_us":70,"threads_started":1}
I20260928 09:03:06.070529 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.066323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.98.3:45881)
0928 09:03:06.066507 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.066512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.066524 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:03:06.066566 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.066568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.066584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.066696 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:06.066885 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.068005 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.069245 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.069935 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.070006 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.070097 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.070394 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.070436 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 09:03:06.070654 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.066168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.0.98.1:36509)
0928 09:03:06.066810 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.066813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.066820 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:03:06.066871 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.066874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.066888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.066987 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:06.067201 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.068212 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.069205 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.069935 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.070007 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.070096 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.070525 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.070570 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":78,"threads_started":1}
I20260928 09:03:06.070706 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.066083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45881 (local address 127.0.0.1:48680)
0928 09:03:06.066328 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.066337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:06.066377 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:06.066713 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:06.066714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:06.066715 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:06.066805 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.066807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.068089 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.068090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.069185 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.069187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.070134 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.070136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.070236 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.070242 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 09:03:06.070294 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:03:06.070348 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":9,"thread_start_us":93,"threads_started":1}
I20260928 09:03:06.071503 18354 tablet_service.cc:1467] Processing CreateTablet for tablet e5800f08332549a3b71196eb21805d6c (DEFAULT_TABLE table=test-table [id=6075e7ce03c7425e933c7e0b4a934653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:06.071602 18223 tablet_service.cc:1467] Processing CreateTablet for tablet e5800f08332549a3b71196eb21805d6c (DEFAULT_TABLE table=test-table [id=6075e7ce03c7425e933c7e0b4a934653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:06.071612 18092 tablet_service.cc:1467] Processing CreateTablet for tablet e5800f08332549a3b71196eb21805d6c (DEFAULT_TABLE table=test-table [id=6075e7ce03c7425e933c7e0b4a934653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:06.071893 18223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5800f08332549a3b71196eb21805d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:06.071893 18354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5800f08332549a3b71196eb21805d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:06.071964 18092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5800f08332549a3b71196eb21805d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:06.074846 18434 tablet_bootstrap.cc:492] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Bootstrap starting.
I20260928 09:03:06.075251 18435 tablet_bootstrap.cc:492] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Bootstrap starting.
I20260928 09:03:06.075506 18436 tablet_bootstrap.cc:492] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Bootstrap starting.
I20260928 09:03:06.075740 18434 tablet_bootstrap.cc:654] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:06.075872 18435 tablet_bootstrap.cc:654] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:06.076085 18434 log.cc:824] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:06.076135 18436 tablet_bootstrap.cc:654] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:06.076193 18435 log.cc:824] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:06.076483 18436 log.cc:824] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:06.077113 18435 tablet_bootstrap.cc:492] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: No bootstrap required, opened a new log
I20260928 09:03:06.077173 18434 tablet_bootstrap.cc:492] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: No bootstrap required, opened a new log
I20260928 09:03:06.077193 18435 ts_tablet_manager.cc:1397] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:03:06.077224 18434 ts_tablet_manager.cc:1397] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:03:06.079859 18434 raft_consensus.cc:348] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.079857 18435 raft_consensus.cc:348] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.080034 18434 raft_consensus.cc:374] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:06.080034 18435 raft_consensus.cc:374] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:06.080072 18434 raft_consensus.cc:729] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71980253e0d34184941add43d1c41980, State: Initialized, Role: FOLLOWER
I20260928 09:03:06.080072 18435 raft_consensus.cc:729] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6117730fab14d62bcc827dd48c96bb5, State: Initialized, Role: FOLLOWER
I20260928 09:03:06.080209 18434 consensus_queue.cc:260] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.080212 18435 consensus_queue.cc:260] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.080480 18435 ts_tablet_manager.cc:1428] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:03:06.080480 18434 ts_tablet_manager.cc:1428] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:03:06.080533 18420 heartbeater.cc:503] Master 127.0.98.62:44729 was elected leader, sending a full tablet report...
I20260928 09:03:06.080603 18158 heartbeater.cc:503] Master 127.0.98.62:44729 was elected leader, sending a full tablet report...
I20260928 09:03:06.081346 18436 tablet_bootstrap.cc:492] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: No bootstrap required, opened a new log
I20260928 09:03:06.081421 18436 ts_tablet_manager.cc:1397] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 09:03:06.083442 18436 raft_consensus.cc:348] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.083560 18436 raft_consensus.cc:374] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:06.083580 18436 raft_consensus.cc:729] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15e84a5f1974dcab38b7f352c0299ca, State: Initialized, Role: FOLLOWER
I20260928 09:03:06.083675 18436 consensus_queue.cc:260] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.083899 18436 ts_tablet_manager.cc:1428] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:06.083995 18289 heartbeater.cc:503] Master 127.0.98.62:44729 was elected leader, sending a full tablet report...
W20260928 09:03:06.170521 18290 tablet.cc:2369] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:06.291715 18440 raft_consensus.cc:482] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260928 09:03:06.291741 18421 tablet.cc:2369] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:06.291836 18440 raft_consensus.cc:504] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.292244 18440 leader_election.cc:288] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f15e84a5f1974dcab38b7f352c0299ca (127.0.98.2:41185), e6117730fab14d62bcc827dd48c96bb5 (127.0.98.3:45881)
I20260928 09:03:06.296370 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.292485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41047 (local address 127.0.98.2:41185)
0928 09:03:06.292533 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.292534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.292548 (+ 14us) server_negotiation.cc:407] Connection header received
0928 09:03:06.292704 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.292705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.292713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.292787 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:06.293096 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.293829 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.295396 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.295953 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.296031 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.296049 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.296257 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.296298 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:03:06.296370 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.292540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59179 (local address 127.0.98.3:45881)
0928 09:03:06.292582 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.292584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.292753 (+ 169us) server_negotiation.cc:407] Connection header received
0928 09:03:06.292786 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.292787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.292796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.292853 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:06.293075 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.293829 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.295390 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.295952 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.296031 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.296041 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.296261 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.296298 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:03:06.296388 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.292420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41185 (local address 127.0.98.1:41047)
0928 09:03:06.292500 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.292512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:06.292559 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:06.292806 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:06.292807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:06.292809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:06.292968 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.292971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.293925 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.293926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.295286 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.295290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.296093 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.296093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.296166 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.296170 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:06.296241 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:03:06.296287 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":87}
I20260928 09:03:06.296404 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.292471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45881 (local address 127.0.98.1:59179)
0928 09:03:06.292665 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.292674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:06.292732 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:06.292894 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:06.292895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:06.292896 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:06.292994 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.292995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.293940 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.293940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.295289 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:06.295290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.296072 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.296073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:06.296171 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.296174 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:06.296241 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:03:06.296286 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":99,"thread_start_us":102,"threads_started":1}
I20260928 09:03:06.296883 18243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5800f08332549a3b71196eb21805d6c" candidate_uuid: "71980253e0d34184941add43d1c41980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15e84a5f1974dcab38b7f352c0299ca" is_pre_election: true
W20260928 09:03:06.297052 18159 tablet.cc:2369] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:06.297070 18243 raft_consensus.cc:2454] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71980253e0d34184941add43d1c41980 in term 0.
I20260928 09:03:06.297096 18374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5800f08332549a3b71196eb21805d6c" candidate_uuid: "71980253e0d34184941add43d1c41980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6117730fab14d62bcc827dd48c96bb5" is_pre_election: true
I20260928 09:03:06.297253 18374 raft_consensus.cc:2454] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71980253e0d34184941add43d1c41980 in term 0.
I20260928 09:03:06.297340 18048 leader_election.cc:302] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71980253e0d34184941add43d1c41980, f15e84a5f1974dcab38b7f352c0299ca; no voters:
I20260928 09:03:06.297468 18440 raft_consensus.cc:2781] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:03:06.297518 18440 raft_consensus.cc:482] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:03:06.297539 18440 raft_consensus.cc:3037] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:06.298269 18440 raft_consensus.cc:504] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.298437 18440 leader_election.cc:288] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [CANDIDATE]: Term 1 election: Requested vote from peers f15e84a5f1974dcab38b7f352c0299ca (127.0.98.2:41185), e6117730fab14d62bcc827dd48c96bb5 (127.0.98.3:45881)
I20260928 09:03:06.298617 18243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5800f08332549a3b71196eb21805d6c" candidate_uuid: "71980253e0d34184941add43d1c41980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15e84a5f1974dcab38b7f352c0299ca"
I20260928 09:03:06.298643 18374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5800f08332549a3b71196eb21805d6c" candidate_uuid: "71980253e0d34184941add43d1c41980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6117730fab14d62bcc827dd48c96bb5"
I20260928 09:03:06.298713 18374 raft_consensus.cc:3037] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:06.298708 18243 raft_consensus.cc:3037] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:06.299672 18243 raft_consensus.cc:2454] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71980253e0d34184941add43d1c41980 in term 1.
I20260928 09:03:06.299669 18374 raft_consensus.cc:2454] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71980253e0d34184941add43d1c41980 in term 1.
I20260928 09:03:06.299882 18046 leader_election.cc:302] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71980253e0d34184941add43d1c41980, e6117730fab14d62bcc827dd48c96bb5; no voters:
I20260928 09:03:06.299996 18440 raft_consensus.cc:2781] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:06.300148 18440 raft_consensus.cc:686] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 1 LEADER]: Becoming Leader. State: Replica: 71980253e0d34184941add43d1c41980, State: Running, Role: LEADER
I20260928 09:03:06.300262 18440 consensus_queue.cc:237] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } }
I20260928 09:03:06.301316 17969 catalog_manager.cc:5495] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71980253e0d34184941add43d1c41980 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "71980253e0d34184941add43d1c41980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71980253e0d34184941add43d1c41980" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 } health_report { overall_health: UNKNOWN } } }
I20260928 09:03:06.354987 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.352549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.0.98.1:36509)
0928 09:03:06.352616 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:06.352618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:06.352659 (+ 41us) server_negotiation.cc:407] Connection header received
0928 09:03:06.352744 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:06.352745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:06.352765 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:06.352825 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:06.352951 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.353414 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.354201 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:06.354383 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:06.354435 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:06.354629 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:06.354729 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:06.354791 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:06.354830 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 09:03:06.354847 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:03:06.356922 18243 raft_consensus.cc:1261] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 1 FOLLOWER]: Refusing update from remote peer 71980253e0d34184941add43d1c41980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:06.356936 18374 raft_consensus.cc:1261] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 1 FOLLOWER]: Refusing update from remote peer 71980253e0d34184941add43d1c41980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:06.357290 18440 consensus_queue.cc:1035] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15e84a5f1974dcab38b7f352c0299ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:06.357430 18444 consensus_queue.cc:1035] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6117730fab14d62bcc827dd48c96bb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:06.360003 18449 mvcc.cc:205] Tried to move back new op lower bound from 7334241019315036160 to 7334241019086745600. Current Snapshot: MvccSnapshot[applied={T|T < 7334241019315036160}]
I20260928 09:03:06.360198 18448 mvcc.cc:205] Tried to move back new op lower bound from 7334241019315036160 to 7334241019086745600. Current Snapshot: MvccSnapshot[applied={T|T < 7334241019315036160}]
I20260928 09:03:06.362633 17969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:06.362720 17969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:06.363481 17969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aaf3932de684c008ad84c9499bb2df6: Sending DeleteTablet for 3 replicas of tablet e5800f08332549a3b71196eb21805d6c
I20260928 09:03:06.363780 18092 tablet_service.cc:1514] Processing DeleteTablet for tablet e5800f08332549a3b71196eb21805d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:06 UTC) from {username='slave'} at 127.0.0.1:45296
I20260928 09:03:06.363834 18223 tablet_service.cc:1514] Processing DeleteTablet for tablet e5800f08332549a3b71196eb21805d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:06 UTC) from {username='slave'} at 127.0.0.1:47362
I20260928 09:03:06.363996 18457 tablet_replica.cc:330] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: stopping tablet replica
I20260928 09:03:06.364050 18354 tablet_service.cc:1514] Processing DeleteTablet for tablet e5800f08332549a3b71196eb21805d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:06 UTC) from {username='slave'} at 127.0.0.1:48680
I20260928 09:03:06.364089 18457 raft_consensus.cc:2229] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:03:06.364203 18457 raft_consensus.cc:2258] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:06.364408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18030
I20260928 09:03:06.364519 18458 tablet_replica.cc:330] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: stopping tablet replica
I20260928 09:03:06.364604 18459 tablet_replica.cc:330] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: stopping tablet replica
I20260928 09:03:06.364674 18457 ts_tablet_manager.cc:1905] T e5800f08332549a3b71196eb21805d6c P 71980253e0d34184941add43d1c41980: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:06.364785 18459 raft_consensus.cc:2229] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:06.364859 18458 raft_consensus.cc:2229] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:06.364974 18458 raft_consensus.cc:2258] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:06.365092 18459 raft_consensus.cc:2258] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:06.365610 18459 ts_tablet_manager.cc:1905] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:06.365635 18458 ts_tablet_manager.cc:1905] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:06.367758 18458 ts_tablet_manager.cc:1918] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:06.367794 18459 ts_tablet_manager.cc:1918] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:06.367861 18458 log.cc:1196] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/e5800f08332549a3b71196eb21805d6c
I20260928 09:03:06.367861 18459 log.cc:1196] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/e5800f08332549a3b71196eb21805d6c
I20260928 09:03:06.368135 18458 ts_tablet_manager.cc:1939] T e5800f08332549a3b71196eb21805d6c P f15e84a5f1974dcab38b7f352c0299ca: Deleting consensus metadata
I20260928 09:03:06.368132 18459 ts_tablet_manager.cc:1939] T e5800f08332549a3b71196eb21805d6c P e6117730fab14d62bcc827dd48c96bb5: Deleting consensus metadata
I20260928 09:03:06.368537 17955 catalog_manager.cc:4841] TS e6117730fab14d62bcc827dd48c96bb5 (127.0.98.3:45881): tablet e5800f08332549a3b71196eb21805d6c (table test-table [id=6075e7ce03c7425e933c7e0b4a934653]) successfully deleted
I20260928 09:03:06.368548 17957 catalog_manager.cc:4841] TS f15e84a5f1974dcab38b7f352c0299ca (127.0.98.2:41185): tablet e5800f08332549a3b71196eb21805d6c (table test-table [id=6075e7ce03c7425e933c7e0b4a934653]) successfully deleted
W20260928 09:03:06.370575 17956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36509 (error 108)
I20260928 09:03:06.371060 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:06.370918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36509 (local address 127.0.0.1:45316)
0928 09:03:06.370972 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:06.371033 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:03:06.371068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18161
W20260928 09:03:06.371174 17956 catalog_manager.cc:4584] TS 71980253e0d34184941add43d1c41980 (127.0.98.1:36509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5800f08332549a3b71196eb21805d6c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36509: connect: Connection refused (error 111)
I20260928 09:03:06.375779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18292
I20260928 09:03:06.380144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 17939
2026-09-28T09:03:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (871 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:06 dist-test-slave-286n krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:06 dist-test-slave-286n krb5kdc[18463](info): set up 2 sockets
Sep 28 09:03:06 dist-test-slave-286n krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260928 09:03:07.424427 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:07 dist-test-slave-286n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:07.547303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:07.627694 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:07.627899 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:07.627923 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:07.629134 18476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:07.629177 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:07.629192 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:07.629204 18476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:07.629261 18476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:07.630697 18476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:07.630967 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:07.631243 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:07.636116 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:07.636157 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:07.636116 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:07.636387 18476 server_base.cc:1034] running on GCE node
I20260928 09:03:07.636775 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:07.637070 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:07.638234 18476 hybrid_clock.cc:648] HybridClock initialized: now 1790586187638225 us; error 55 us; skew 500 ppm
Sep 28 09:03:07 dist-test-slave-286n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:07.640617 18476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:07.641094 18476 webserver.cc:460] Webserver started at http://127.0.98.62:35469/ using document root <none> and password file <none>
I20260928 09:03:07.641322 18476 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:07.641450 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:07.641618 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:07.642669 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "54222d360f1d4c58b160ba35be042f8e"
format_stamp: "Formatted at 2026-09-28 09:03:07 on dist-test-slave-286n"
server_key: "9e1f49749dbe5dd2c90e66515d1aac28"
server_key_iv: "8c96d1aa0435f961ea0c33148adf51d7"
server_key_version: "encryptionkey@0"
I20260928 09:03:07.643028 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "54222d360f1d4c58b160ba35be042f8e"
format_stamp: "Formatted at 2026-09-28 09:03:07 on dist-test-slave-286n"
server_key: "9e1f49749dbe5dd2c90e66515d1aac28"
server_key_iv: "8c96d1aa0435f961ea0c33148adf51d7"
server_key_version: "encryptionkey@0"
I20260928 09:03:07.644398 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 09:03:07.645190 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:07.645367 18476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:07.645466 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "54222d360f1d4c58b160ba35be042f8e"
format_stamp: "Formatted at 2026-09-28 09:03:07 on dist-test-slave-286n"
server_key: "9e1f49749dbe5dd2c90e66515d1aac28"
server_key_iv: "8c96d1aa0435f961ea0c33148adf51d7"
server_key_version: "encryptionkey@0"
I20260928 09:03:07.645553 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:07.665426 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:07.667228 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:07.667399 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:07.671545 18476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40235
I20260928 09:03:07.671595 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40235 every 8 connection(s)
I20260928 09:03:07.671976 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:07.672974 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:07.675206 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e: Bootstrap starting.
I20260928 09:03:07.675719 18544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:07.675967 18544 log.cc:824] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:07.676617 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e: No bootstrap required, opened a new log
I20260928 09:03:07.678224 18544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54222d360f1d4c58b160ba35be042f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 09:03:07.678362 18544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:07.678385 18544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54222d360f1d4c58b160ba35be042f8e, State: Initialized, Role: FOLLOWER
I20260928 09:03:07.678539 18544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54222d360f1d4c58b160ba35be042f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 09:03:07.678618 18544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:07.678640 18544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:07.678668 18544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:07.679140 18544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54222d360f1d4c58b160ba35be042f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 09:03:07.679224 18544 leader_election.cc:302] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54222d360f1d4c58b160ba35be042f8e; no voters:
I20260928 09:03:07.679374 18544 leader_election.cc:288] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:07.679406 18547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:07.679530 18547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [term 1 LEADER]: Becoming Leader. State: Replica: 54222d360f1d4c58b160ba35be042f8e, State: Running, Role: LEADER
I20260928 09:03:07.679607 18547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54222d360f1d4c58b160ba35be042f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 09:03:07.679636 18544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:07.680088 18548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54222d360f1d4c58b160ba35be042f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54222d360f1d4c58b160ba35be042f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } } }
I20260928 09:03:07.680174 18548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:07.680301 18549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54222d360f1d4c58b160ba35be042f8e. Latest consensus state: current_term: 1 leader_uuid: "54222d360f1d4c58b160ba35be042f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54222d360f1d4c58b160ba35be042f8e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } } }
I20260928 09:03:07.680433 18549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:07.680799 18552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:07.681324 18552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:07.682209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18476
I20260928 09:03:07.682349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:07.682570 392 external_mini_cluster.cc:1428] Setting key b435635eb79477f8e3244c7b77308602
I20260928 09:03:07.683259 18552 catalog_manager.cc:1347] Generated new cluster ID: 2c01cfdf5fde47efaa2f6c6a5dbb4498
I20260928 09:03:07.683312 18552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:07 dist-test-slave-286n krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:07.695577 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:07.682993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.0.98.62:40235)
0928 09:03:07.684772 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:07.684782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:07.684856 (+ 74us) server_negotiation.cc:407] Connection header received
0928 09:03:07.684984 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:07.684991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:07.685055 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:07.685216 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:07.685915 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:07.687371 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:07.688393 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:07.689046 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:07.689157 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:07.691969 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:07.691973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:07.691973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:07.691986 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:07.693962 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:07.694362 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:07.694365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:07.694365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:07.694468 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:07.694675 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:07.694675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:07.694675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:07.694962 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:07.695093 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:07.695215 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 09:03:07.695341 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1723,"thread_start_us":81,"threads_started":1}
I20260928 09:03:07.705328 18552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:07.706099 18552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:07.712188 18552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e: Generated new TSK 0
I20260928 09:03:07.712426 18552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:03:07.741778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:07.850247 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:07.850455 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:07.850481 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:07.851569 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:07.851604 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:07.851630 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:03:07.853111 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:07.853296 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:07.853466 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:07.858326 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:07.858316 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:07.858496 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:07.858624 18570 server_base.cc:1034] running on GCE node
I20260928 09:03:07.858819 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:07.859010 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:07.860149 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790586187860136 us; error 27 us; skew 500 ppm
Sep 28 09:03:07 dist-test-slave-286n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:07.862747 18570 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:03:07.863183 18570 webserver.cc:460] Webserver started at http://127.0.98.1:37709/ using document root <none> and password file <none>
I20260928 09:03:07.863418 18570 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:07.863463 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:07.863613 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:07.864773 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "66535a03de7f4e9b82947865a0fdb7d1"
format_stamp: "Formatted at 2026-09-28 09:03:07 on dist-test-slave-286n"
server_key: "819833e383c88b95c8f060e1aef3d686"
server_key_iv: "4456aa158745cd9a280d8bb9da3ed3cf"
server_key_version: "encryptionkey@0"
I20260928 09:03:07.865127 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66535a03de7f4e9b82947865a0fdb7d1"
format_stamp: "Formatted at 2026-09-28 09:03:07 on dist-test-slave-286n"
server_key: "819833e383c88b95c8f060e1aef3d686"
server_key_iv: "4456aa158745cd9a280d8bb9da3ed3cf"
server_key_version: "encryptionkey@0"
I20260928 09:03:07.866400 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:07.867112 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:07.867324 18570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:07.867431 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "66535a03de7f4e9b82947865a0fdb7d1"
format_stamp: "Formatted at 2026-09-28 09:03:07 on dist-test-slave-286n"
server_key: "819833e383c88b95c8f060e1aef3d686"
server_key_iv: "4456aa158745cd9a280d8bb9da3ed3cf"
server_key_version: "encryptionkey@0"
I20260928 09:03:07.867503 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:07.892103 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:07.894156 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:07.894330 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:07.894568 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:07.894891 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:07.894951 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:07.894992 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:07.895021 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:07.901081 18570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43927
I20260928 09:03:07.901134 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43927 every 8 connection(s)
I20260928 09:03:07.901540 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:03:07.907853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18570
I20260928 09:03:07.907977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
Sep 28 09:03:07 dist-test-slave-286n krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:07.908207 392 external_mini_cluster.cc:1428] Setting key abb219c9a9e2a1bfe2da4acb84d9fcac
I20260928 09:03:07.912076 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:07.902314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53437 (local address 127.0.98.62:40235)
0928 09:03:07.902403 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:07.902406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:07.902580 (+ 174us) server_negotiation.cc:407] Connection header received
0928 09:03:07.903586 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:07.903587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:07.903600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:07.903653 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:07.904294 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:07.904908 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:07.905905 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:07.906124 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:07.906204 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:07.908721 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:07.908722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:07.908722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:07.908730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:07.910440 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:07.910974 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:07.910975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:07.910976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:07.911029 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:07.911314 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:07.911315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:07.911315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:07.911416 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:07.911466 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:07.911834 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 09:03:07.911930 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 09:03:07.912111 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:07.902263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.1:53437)
0928 09:03:07.902462 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:07.902498 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:07.903474 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:07.903739 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:07.903743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:07.904004 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:07.904195 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:07.904199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:07.905000 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:07.905001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:07.905812 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:07.905814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:07.906219 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:07.906220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:07.906341 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:07.906843 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:07.906851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:07.908580 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:07.910548 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:07.910556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:07.910559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:07.910892 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:07.911096 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:07.911097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:07.911097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:07.911249 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:07.911480 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:07.911486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:07.911662 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 09:03:07.911810 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 09:03:07.911951 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260928 09:03:07.912716 18699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 09:03:07.912815 18699 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:07.913010 18699 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 09:03:07.913611 18508 ts_manager.cc:194] Registered new tserver with Master: 66535a03de7f4e9b82947865a0fdb7d1 (127.0.98.1:43927)
I20260928 09:03:07.914659 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53437
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:03:07.933389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:08.015394 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:08.015619 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:08.015646 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:08.016798 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:08.016837 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:08.016865 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:03:08.018292 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:08.018514 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:08.018709 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:08.023861 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.023893 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.023911 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.024156 18704 server_base.cc:1034] running on GCE node
I20260928 09:03:08.024430 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:08.024720 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:08.025897 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790586188025863 us; error 46 us; skew 500 ppm
Sep 28 09:03:08 dist-test-slave-286n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:08.028358 18704 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:03:08.028828 18704 webserver.cc:460] Webserver started at http://127.0.98.2:41937/ using document root <none> and password file <none>
I20260928 09:03:08.029028 18704 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:08.029068 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:08.029155 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:08.030133 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "8790f2bdead2c205a3619026ad414444"
server_key_iv: "da6cfacf33b0e07931606f262b1c86cc"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.030423 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "8790f2bdead2c205a3619026ad414444"
server_key_iv: "da6cfacf33b0e07931606f262b1c86cc"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.031635 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:08.032317 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.032472 18704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:03:08.032554 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "8790f2bdead2c205a3619026ad414444"
server_key_iv: "da6cfacf33b0e07931606f262b1c86cc"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.032609 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:08.041965 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:08.044171 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:08.044278 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:08.044451 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:08.044782 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:08.044817 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.044840 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:08.044853 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.050398 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40341
I20260928 09:03:08.050478 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40341 every 8 connection(s)
I20260928 09:03:08.050777 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:03:08 dist-test-slave-286n krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:08.058348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18704
I20260928 09:03:08.058470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:03:08.058683 392 external_mini_cluster.cc:1428] Setting key adbad897c0f8e82f894bba0c876b6e6e
I20260928 09:03:08.060467 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.051553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34523 (local address 127.0.98.62:40235)
0928 09:03:08.051632 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.051634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.051866 (+ 232us) server_negotiation.cc:407] Connection header received
0928 09:03:08.052855 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.052856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.052867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.052920 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.053515 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.054019 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.054994 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.055177 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.055229 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.057449 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.057451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.057451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:08.057457 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.058921 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:08.059457 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:08.059459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:08.059460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:08.059522 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:08.059758 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:08.059759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:08.059759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:08.059903 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.059974 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.060221 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.060320 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 09:03:08.060479 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.051552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.2:34523)
0928 09:03:08.051746 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.051781 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.052772 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.052973 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.052977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.053229 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:08.053414 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.053418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.054102 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.054102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.054920 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.054921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.055251 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.055252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.055353 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.055928 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:08.055936 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:08.057356 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:08.059052 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:08.059059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:08.059061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:08.059379 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:08.059588 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:08.059589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:08.059590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:08.059691 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:08.059967 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:08.059971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:08.060081 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.060176 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.060294 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260928 09:03:08.061614 18833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 09:03:08.061753 18833 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:08.061961 18833 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 09:03:08.062407 18508 ts_manager.cc:194] Registered new tserver with Master: ae5d6a4cdd3b40e3aae7c00687b187cc (127.0.98.2:40341)
I20260928 09:03:08.062839 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34523
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:03:08.085547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:08.166611 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:08.166826 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:08.166848 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:08.167961 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:08.167999 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:08.168025 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:03:08.169608 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:08.169857 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:08.170097 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:08.175133 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.175197 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.175143 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.175484 18838 server_base.cc:1034] running on GCE node
I20260928 09:03:08.175792 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:08.176029 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:08.177169 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790586188177147 us; error 34 us; skew 500 ppm
Sep 28 09:03:08 dist-test-slave-286n krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:08.179795 18838 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:03:08.180287 18838 webserver.cc:460] Webserver started at http://127.0.98.3:45447/ using document root <none> and password file <none>
I20260928 09:03:08.180529 18838 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:08.180625 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:08.180795 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:08.181915 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "2047381daf4543a8ac6a88483c7f71b5"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d4e4e71ee9f084d1e500b57537612baf"
server_key_iv: "11528583377ff92df65a4425ce6d8b88"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.182276 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2047381daf4543a8ac6a88483c7f71b5"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d4e4e71ee9f084d1e500b57537612baf"
server_key_iv: "11528583377ff92df65a4425ce6d8b88"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.183627 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:08.184337 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.184500 18838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:08.184679 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "2047381daf4543a8ac6a88483c7f71b5"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d4e4e71ee9f084d1e500b57537612baf"
server_key_iv: "11528583377ff92df65a4425ce6d8b88"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.184765 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:08.197819 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:08.200925 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:08.201090 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:08.201359 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:08.201762 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:08.201805 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.201843 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:08.201869 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.208467 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44191
I20260928 09:03:08.208539 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44191 every 8 connection(s)
I20260928 09:03:08.208928 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:03:08.210382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 18838
I20260928 09:03:08.210485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:03:08.210721 392 external_mini_cluster.cc:1428] Setting key fececd34c3daaefbcf2a9f5f1d4b0185
Sep 28 09:03:08 dist-test-slave-286n krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:08.218520 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.209636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42491 (local address 127.0.98.62:40235)
0928 09:03:08.209680 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.209682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.209900 (+ 218us) server_negotiation.cc:407] Connection header received
0928 09:03:08.211210 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.211211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.211225 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.211290 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.212101 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.212740 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.213720 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.213903 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.213957 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.215904 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.215905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.215905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:08.215911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.217379 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:08.217758 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:08.217759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:08.217760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:08.217806 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:08.218007 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:08.218008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:08.218008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:08.218106 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.218154 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.218396 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.218455 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:03:08.218701 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.209620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.3:42491)
0928 09:03:08.209799 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.209834 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.211056 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.211390 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.211395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.211779 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:08.211998 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.212002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.212824 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.212825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.213646 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.213648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.213975 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.213976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.214070 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.214526 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:08.214534 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:08.215826 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:08.217460 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:08.217466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:08.217468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:08.217695 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:08.217862 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:08.217863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:08.217863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:08.217956 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:08.218159 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:08.218163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:08.218291 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.218372 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.218486 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
I20260928 09:03:08.219172 18967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 09:03:08.219262 18967 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:08.219401 18967 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 09:03:08.219722 18508 ts_manager.cc:194] Registered new tserver with Master: 2047381daf4543a8ac6a88483c7f71b5 (127.0.98.3:44191)
I20260928 09:03:08.220044 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42491
I20260928 09:03:08.222450 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:03:08.230017 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.223733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.0.98.62:40235)
0928 09:03:08.223815 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.223817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.223909 (+ 92us) server_negotiation.cc:407] Connection header received
0928 09:03:08.223986 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.223987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.223996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.224068 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.224608 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.225110 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.225855 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.226049 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.226136 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.227375 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.227376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.227377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:08.227384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.228758 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:08.229138 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:08.229139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:08.229140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:08.229200 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:08.229414 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:08.229415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:08.229415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:08.229555 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.229675 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.229852 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.229919 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 09:03:08.231644 18508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:03:08.233349 18508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:03:08.240299 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.236031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.0.98.1:43927)
0928 09:03:08.236304 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.236307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.236345 (+ 38us) server_negotiation.cc:407] Connection header received
0928 09:03:08.236460 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.236462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.236476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.236539 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.236865 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.237944 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.239374 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.239899 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.240008 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.240068 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.240175 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.240219 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
I20260928 09:03:08.240291 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.235840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43927 (local address 127.0.0.1:33660)
0928 09:03:08.236221 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.236266 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.236372 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.236743 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.236743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.236745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.236801 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.236803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.238049 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.238050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.239300 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.239301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.240012 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.240013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.240081 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.240086 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.240149 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.240194 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":43,"thread_start_us":49,"threads_started":1}
I20260928 09:03:08.240360 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.236044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.0.98.3:44191)
0928 09:03:08.236256 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.236260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.236346 (+ 86us) server_negotiation.cc:407] Connection header received
0928 09:03:08.236454 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.236457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.236473 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.236558 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.236834 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.237942 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.239097 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.239761 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.239831 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.239889 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.240243 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.240279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
I20260928 09:03:08.240401 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.235853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44191 (local address 127.0.0.1:37326)
0928 09:03:08.236272 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.236282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.236353 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.236669 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.236672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.236674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.236747 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.236749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.238073 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.238074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.239013 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.239015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.239924 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.239925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.240002 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.240018 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.240092 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.240145 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":77,"threads_started":1}
I20260928 09:03:08.240470 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.236029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.0.98.2:40341)
0928 09:03:08.236220 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.236224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.236571 (+ 347us) server_negotiation.cc:407] Connection header received
0928 09:03:08.236620 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.236623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.236660 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.236740 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.236930 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.237668 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.239300 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.239985 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.240048 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.240102 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.240339 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.240388 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260928 09:03:08.240713 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.235843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40341 (local address 127.0.0.1:33846)
0928 09:03:08.236230 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.236462 (+ 232us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.236507 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.236830 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.236831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.236833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.236872 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.236874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.237786 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.237788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.239209 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.239212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.240135 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.240135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.240187 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.240191 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.240236 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.240272 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":88,"threads_started":1}
I20260928 09:03:08.241549 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 217f7b0cf5d44ee495d5dab4f46501a5 (DEFAULT_TABLE table=test-table [id=e77e46a60c1342eb9f9c067e9fad33c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:08.241559 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 217f7b0cf5d44ee495d5dab4f46501a5 (DEFAULT_TABLE table=test-table [id=e77e46a60c1342eb9f9c067e9fad33c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:08.241817 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f7b0cf5d44ee495d5dab4f46501a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.241815 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f7b0cf5d44ee495d5dab4f46501a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.244408 18984 tablet_bootstrap.cc:492] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Bootstrap starting.
I20260928 09:03:08.244537 18985 tablet_bootstrap.cc:492] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Bootstrap starting.
I20260928 09:03:08.245098 18984 tablet_bootstrap.cc:654] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.245245 18985 tablet_bootstrap.cc:654] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.245311 18633 tablet_service.cc:1467] Processing CreateTablet for tablet 217f7b0cf5d44ee495d5dab4f46501a5 (DEFAULT_TABLE table=test-table [id=e77e46a60c1342eb9f9c067e9fad33c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:08.245664 18633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217f7b0cf5d44ee495d5dab4f46501a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.245671 18984 log.cc:824] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.245921 18985 log.cc:824] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.246449 18984 tablet_bootstrap.cc:492] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: No bootstrap required, opened a new log
I20260928 09:03:08.246527 18984 ts_tablet_manager.cc:1397] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:08.246721 18985 tablet_bootstrap.cc:492] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: No bootstrap required, opened a new log
I20260928 09:03:08.246790 18985 ts_tablet_manager.cc:1397] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:08.248761 18988 tablet_bootstrap.cc:492] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1: Bootstrap starting.
I20260928 09:03:08.248862 18984 raft_consensus.cc:348] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.248920 18985 raft_consensus.cc:348] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.249078 18985 raft_consensus.cc:374] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.249078 18984 raft_consensus.cc:374] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.249118 18985 raft_consensus.cc:729] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2047381daf4543a8ac6a88483c7f71b5, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.249118 18984 raft_consensus.cc:729] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5d6a4cdd3b40e3aae7c00687b187cc, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.249246 18985 consensus_queue.cc:260] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.249276 18984 consensus_queue.cc:260] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.249497 18988 tablet_bootstrap.cc:654] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.249557 18833 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 09:03:08.249563 18985 ts_tablet_manager.cc:1428] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:03:08.249686 18967 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 09:03:08.249864 18984 ts_tablet_manager.cc:1428] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:03:08.250326 18988 log.cc:824] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.251194 18988 tablet_bootstrap.cc:492] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1: No bootstrap required, opened a new log
I20260928 09:03:08.251274 18988 ts_tablet_manager.cc:1397] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:03:08.252837 18988 raft_consensus.cc:348] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.252990 18988 raft_consensus.cc:374] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.253036 18988 raft_consensus.cc:729] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66535a03de7f4e9b82947865a0fdb7d1, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.253170 18988 consensus_queue.cc:260] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.253417 18988 ts_tablet_manager.cc:1428] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:08.253464 18699 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 09:03:08.293725 18992 raft_consensus.cc:482] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:03:08.293905 18992 raft_consensus.cc:504] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.294528 18992 leader_election.cc:288] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5d6a4cdd3b40e3aae7c00687b187cc (127.0.98.2:40341), 2047381daf4543a8ac6a88483c7f71b5 (127.0.98.3:44191)
I20260928 09:03:08.298529 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.294831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51287 (local address 127.0.98.3:44191)
0928 09:03:08.294874 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.294876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.295046 (+ 170us) server_negotiation.cc:407] Connection header received
0928 09:03:08.295081 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.295082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.295095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.295178 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.295330 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.296011 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.297578 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.298151 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.298238 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.298248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.298408 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.298456 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:03:08.298542 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.294700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40341 (local address 127.0.98.1:32957)
0928 09:03:08.294744 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.294756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.294802 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.295011 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.295013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.295015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.295087 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.295089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.296044 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.296045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.297433 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.297436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.298237 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.298238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.298328 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.298330 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.298382 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.298437 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":84}
I20260928 09:03:08.298590 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.294784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32957 (local address 127.0.98.2:40341)
0928 09:03:08.294834 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.294836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.294847 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:03:08.294878 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.294879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.294894 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.294962 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.295176 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.295962 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.297551 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.298105 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.298221 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.298234 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.298408 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.298511 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:08.298823 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.294717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44191 (local address 127.0.98.1:51287)
0928 09:03:08.294921 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.294929 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.294999 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.295196 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.295196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.295197 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.295260 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.295262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.296093 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.296093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.297433 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.297436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.298244 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.298245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.298317 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.298321 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.298382 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.298437 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":49,"thread_start_us":131,"threads_started":1}
I20260928 09:03:08.299009 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f7b0cf5d44ee495d5dab4f46501a5" candidate_uuid: "66535a03de7f4e9b82947865a0fdb7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" is_pre_election: true
I20260928 09:03:08.299196 18787 raft_consensus.cc:2454] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66535a03de7f4e9b82947865a0fdb7d1 in term 0.
I20260928 09:03:08.299182 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f7b0cf5d44ee495d5dab4f46501a5" candidate_uuid: "66535a03de7f4e9b82947865a0fdb7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2047381daf4543a8ac6a88483c7f71b5" is_pre_election: true
I20260928 09:03:08.299351 18921 raft_consensus.cc:2454] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66535a03de7f4e9b82947865a0fdb7d1 in term 0.
I20260928 09:03:08.299481 18587 leader_election.cc:302] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66535a03de7f4e9b82947865a0fdb7d1, ae5d6a4cdd3b40e3aae7c00687b187cc; no voters:
I20260928 09:03:08.299620 18992 raft_consensus.cc:2781] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:03:08.299680 18992 raft_consensus.cc:482] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:03:08.299717 18992 raft_consensus.cc:3037] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:08.300447 18992 raft_consensus.cc:504] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.300606 18992 leader_election.cc:288] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [CANDIDATE]: Term 1 election: Requested vote from peers ae5d6a4cdd3b40e3aae7c00687b187cc (127.0.98.2:40341), 2047381daf4543a8ac6a88483c7f71b5 (127.0.98.3:44191)
I20260928 09:03:08.300799 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f7b0cf5d44ee495d5dab4f46501a5" candidate_uuid: "66535a03de7f4e9b82947865a0fdb7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc"
I20260928 09:03:08.300890 18787 raft_consensus.cc:3037] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 0 FOLLOWER]: Advancing to term 1
W20260928 09:03:08.301327 18834 tablet.cc:2369] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:08.301836 18787 raft_consensus.cc:2454] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66535a03de7f4e9b82947865a0fdb7d1 in term 1.
I20260928 09:03:08.301967 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217f7b0cf5d44ee495d5dab4f46501a5" candidate_uuid: "66535a03de7f4e9b82947865a0fdb7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2047381daf4543a8ac6a88483c7f71b5"
I20260928 09:03:08.302013 18587 leader_election.cc:302] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66535a03de7f4e9b82947865a0fdb7d1, ae5d6a4cdd3b40e3aae7c00687b187cc; no voters:
I20260928 09:03:08.302058 18921 raft_consensus.cc:3037] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:08.302094 18992 raft_consensus.cc:2781] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:08.302246 18992 raft_consensus.cc:686] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [term 1 LEADER]: Becoming Leader. State: Replica: 66535a03de7f4e9b82947865a0fdb7d1, State: Running, Role: LEADER
I20260928 09:03:08.302347 18992 consensus_queue.cc:237] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } }
I20260928 09:03:08.303023 18921 raft_consensus.cc:2454] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66535a03de7f4e9b82947865a0fdb7d1 in term 1.
I20260928 09:03:08.303249 18507 catalog_manager.cc:5495] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66535a03de7f4e9b82947865a0fdb7d1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "66535a03de7f4e9b82947865a0fdb7d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66535a03de7f4e9b82947865a0fdb7d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } health_report { overall_health: UNKNOWN } } }
I20260928 09:03:08.314071 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.311652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.0.98.1:43927)
0928 09:03:08.311720 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.311722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.311731 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:03:08.311762 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.311763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.311782 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.311847 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:08.311983 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.312481 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.313263 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.313492 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.313557 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.313697 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:08.313782 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:08.313857 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.313904 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.313941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:03:08.315850 18787 raft_consensus.cc:1261] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 1 FOLLOWER]: Refusing update from remote peer 66535a03de7f4e9b82947865a0fdb7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:08.315832 18921 raft_consensus.cc:1261] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 1 FOLLOWER]: Refusing update from remote peer 66535a03de7f4e9b82947865a0fdb7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:08.316210 18992 consensus_queue.cc:1035] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5d6a4cdd3b40e3aae7c00687b187cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:08.316376 18994 consensus_queue.cc:1035] T 217f7b0cf5d44ee495d5dab4f46501a5 P 66535a03de7f4e9b82947865a0fdb7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2047381daf4543a8ac6a88483c7f71b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:08.321645 18508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:08.321756 18508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:08.322584 18508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54222d360f1d4c58b160ba35be042f8e: Sending DeleteTablet for 3 replicas of tablet 217f7b0cf5d44ee495d5dab4f46501a5
I20260928 09:03:08.322885 18633 tablet_service.cc:1514] Processing DeleteTablet for tablet 217f7b0cf5d44ee495d5dab4f46501a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33660
I20260928 09:03:08.322934 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet 217f7b0cf5d44ee495d5dab4f46501a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37326
I20260928 09:03:08.323060 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 217f7b0cf5d44ee495d5dab4f46501a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33846
I20260928 09:03:08.323293 19008 tablet_replica.cc:330] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: stopping tablet replica
I20260928 09:03:08.323447 19008 raft_consensus.cc:2229] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:08.323566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18570
I20260928 09:03:08.324344 19008 raft_consensus.cc:2258] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:08.324862 19008 ts_tablet_manager.cc:1905] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:08.325583 19007 tablet_replica.cc:330] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: stopping tablet replica
I20260928 09:03:08.325687 19007 raft_consensus.cc:2229] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:08.325788 19007 raft_consensus.cc:2258] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:08.327049 19008 ts_tablet_manager.cc:1918] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:08.327133 19008 log.cc:1196] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/217f7b0cf5d44ee495d5dab4f46501a5
I20260928 09:03:08.327374 19008 ts_tablet_manager.cc:1939] T 217f7b0cf5d44ee495d5dab4f46501a5 P ae5d6a4cdd3b40e3aae7c00687b187cc: Deleting consensus metadata
I20260928 09:03:08.327826 18493 catalog_manager.cc:4841] TS ae5d6a4cdd3b40e3aae7c00687b187cc (127.0.98.2:40341): tablet 217f7b0cf5d44ee495d5dab4f46501a5 (table test-table [id=e77e46a60c1342eb9f9c067e9fad33c3]) successfully deleted
I20260928 09:03:08.329100 19007 ts_tablet_manager.cc:1905] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 09:03:08.329532 18494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43927 (error 108)
W20260928 09:03:08.330034 18980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:08.329912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43927 (local address 127.0.0.1:33676)
0928 09:03:08.329967 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.330003 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43927: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 09:03:08.330158 18494 catalog_manager.cc:4584] TS 66535a03de7f4e9b82947865a0fdb7d1 (127.0.98.1:43927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 217f7b0cf5d44ee495d5dab4f46501a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43927: connect: Connection reset by peer (error 104)
I20260928 09:03:08.330529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18704
I20260928 09:03:08.331349 19007 ts_tablet_manager.cc:1918] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:08.331444 19007 log.cc:1196] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/217f7b0cf5d44ee495d5dab4f46501a5
I20260928 09:03:08.331735 19007 ts_tablet_manager.cc:1939] T 217f7b0cf5d44ee495d5dab4f46501a5 P 2047381daf4543a8ac6a88483c7f71b5: Deleting consensus metadata
I20260928 09:03:08.332365 18492 catalog_manager.cc:4841] TS 2047381daf4543a8ac6a88483c7f71b5 (127.0.98.3:44191): tablet 217f7b0cf5d44ee495d5dab4f46501a5 (table test-table [id=e77e46a60c1342eb9f9c067e9fad33c3]) successfully deleted
I20260928 09:03:08.335489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18838
I20260928 09:03:08.340770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 18476
2026-09-28T09:03:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1961 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T09:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:08Z Disabled control of system clock
I20260928 09:03:08.368583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37177
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:08.448329 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:08.448549 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:08.448570 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:08.449738 19013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:08.449782 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:08.449795 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:08.449808 19013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:08.449820 19013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:08.451089 19013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:08.451282 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:08.451527 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:08.456248 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.456282 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.456300 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.456506 19013 server_base.cc:1034] running on GCE node
I20260928 09:03:08.456813 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:08.457072 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:08.458214 19013 hybrid_clock.cc:648] HybridClock initialized: now 1790586188458203 us; error 30 us; skew 500 ppm
I20260928 09:03:08.459491 19013 webserver.cc:460] Webserver started at http://127.0.98.62:44767/ using document root <none> and password file <none>
I20260928 09:03:08.459774 19013 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:08.459846 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:08.460032 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:08.461283 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "b0e23039bf3740b5a24f62f1c023ad57"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d8ff093fdde5a04008dcd902552161e2"
server_key_iv: "4310f39fab12233d5c004b38c06a7a9e"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.461671 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0e23039bf3740b5a24f62f1c023ad57"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d8ff093fdde5a04008dcd902552161e2"
server_key_iv: "4310f39fab12233d5c004b38c06a7a9e"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.463016 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:08.463701 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.463883 19013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:08.463981 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "b0e23039bf3740b5a24f62f1c023ad57"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d8ff093fdde5a04008dcd902552161e2"
server_key_iv: "4310f39fab12233d5c004b38c06a7a9e"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.464059 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:08.478282 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:08.478614 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:08.478770 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:08.482903 19013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37177
I20260928 09:03:08.482940 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37177 every 8 connection(s)
I20260928 09:03:08.483316 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:08.484395 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.486788 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57: Bootstrap starting.
I20260928 09:03:08.487406 19080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.487707 19080 log.cc:824] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.488463 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57: No bootstrap required, opened a new log
I20260928 09:03:08.490137 19080 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e23039bf3740b5a24f62f1c023ad57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260928 09:03:08.490276 19080 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.490299 19080 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e23039bf3740b5a24f62f1c023ad57, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.490458 19080 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e23039bf3740b5a24f62f1c023ad57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260928 09:03:08.490533 19080 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:08.490554 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:08.490604 19080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:08.491133 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e23039bf3740b5a24f62f1c023ad57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260928 09:03:08.491243 19080 leader_election.cc:302] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e23039bf3740b5a24f62f1c023ad57; no voters:
I20260928 09:03:08.491430 19080 leader_election.cc:288] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:08.491488 19083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:08.491699 19083 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [term 1 LEADER]: Becoming Leader. State: Replica: b0e23039bf3740b5a24f62f1c023ad57, State: Running, Role: LEADER
I20260928 09:03:08.491698 19080 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:08.491813 19083 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e23039bf3740b5a24f62f1c023ad57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260928 09:03:08.492193 19085 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e23039bf3740b5a24f62f1c023ad57. Latest consensus state: current_term: 1 leader_uuid: "b0e23039bf3740b5a24f62f1c023ad57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e23039bf3740b5a24f62f1c023ad57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } } }
I20260928 09:03:08.492288 19085 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:08.492285 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e23039bf3740b5a24f62f1c023ad57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e23039bf3740b5a24f62f1c023ad57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } } }
I20260928 09:03:08.492471 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:08.492560 19092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:08.492986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19013
I20260928 09:03:08.493098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:08.493173 19092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:08.493300 392 external_mini_cluster.cc:1428] Setting key f2d52315f7cf8a6a22f6f3287f0b4bc8
I20260928 09:03:08.494949 19092 catalog_manager.cc:1347] Generated new cluster ID: a0b33519582b424b853150fa98fe6a64
I20260928 09:03:08.495009 19092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:08.499907 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.493925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.0.98.62:37177)
0928 09:03:08.494367 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.494376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.494409 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:03:08.494541 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.494548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.494618 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.494758 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.495027 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.496372 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.497480 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.498095 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.498198 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.499081 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.499084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.499085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:08.499279 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.499321 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:08.499334 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.499380 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.499632 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.499707 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":77,"threads_started":1}
I20260928 09:03:08.511083 19092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:08.511608 19092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:08.519538 19092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57: Generated new TSK 0
I20260928 09:03:08.519729 19092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:08.522214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37177
--builtin_ntp_servers=127.0.98.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:08.605592 19104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:08.605818 19104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:08.605845 19104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:08.607137 19104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:08.607182 19104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:08.607208 19104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:03:08.608592 19104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:08.608908 19104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:08.609239 19104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:08.615114 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.615486 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.615782 19109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.615942 19104 server_base.cc:1034] running on GCE node
I20260928 09:03:08.616211 19104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:08.616546 19104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:08.617679 19104 hybrid_clock.cc:648] HybridClock initialized: now 1790586188617666 us; error 34 us; skew 500 ppm
I20260928 09:03:08.619149 19104 webserver.cc:460] Webserver started at http://127.0.98.1:38499/ using document root <none> and password file <none>
I20260928 09:03:08.619417 19104 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:08.619489 19104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:08.619637 19104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:08.620817 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "013c6c72256340739e034898b13a82a9"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "b6f39bde6a269dd52f4f499990345af0"
server_key_iv: "de65ab5abb9fe33113dc81521bd5794b"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.621186 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "013c6c72256340739e034898b13a82a9"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "b6f39bde6a269dd52f4f499990345af0"
server_key_iv: "de65ab5abb9fe33113dc81521bd5794b"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.622737 19104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:03:08.624285 19118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.624475 19104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 09:03:08.624563 19104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "013c6c72256340739e034898b13a82a9"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "b6f39bde6a269dd52f4f499990345af0"
server_key_iv: "de65ab5abb9fe33113dc81521bd5794b"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.624658 19104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:08.646059 19104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:08.646428 19104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:08.646597 19104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:08.646831 19104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:08.647233 19104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:08.647282 19104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.647320 19104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:08.647351 19104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.655122 19104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36451
I20260928 09:03:08.655189 19231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36451 every 8 connection(s)
I20260928 09:03:08.655630 19104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:03:08.657178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19104
I20260928 09:03:08.657292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:03:08.657495 392 external_mini_cluster.cc:1428] Setting key 9cd9b1f4400cb7ff056563b3ba1e70da
I20260928 09:03:08.658608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37177
--builtin_ntp_servers=127.0.98.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 09:03:08.663200 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.656437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40197 (local address 127.0.98.62:37177)
0928 09:03:08.656512 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.656515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.656852 (+ 337us) server_negotiation.cc:407] Connection header received
0928 09:03:08.657657 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.657658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.657672 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.657736 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.658697 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.659543 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.661303 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.661662 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.661711 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.662871 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.662873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.662873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:08.662878 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.662894 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:08.662899 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.662929 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.663074 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.663109 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 09:03:08.663914 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.656446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37177 (local address 127.0.98.1:40197)
0928 09:03:08.656690 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.656743 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.657546 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.657830 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.657835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.657846 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:08.658597 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.658605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.659762 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.659763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.661168 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.661171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.661815 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.661816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.661999 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.662680 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:08.662681 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:08.662728 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:08.662760 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:08.662976 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:08.662981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:08.662986 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.663456 (+ 470us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.663604 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260928 09:03:08.664454 19232 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37177
I20260928 09:03:08.664572 19232 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:08.664829 19232 heartbeater.cc:511] Master 127.0.98.62:37177 requested a full tablet report, sending...
I20260928 09:03:08.665412 19044 ts_manager.cc:194] Registered new tserver with Master: 013c6c72256340739e034898b13a82a9 (127.0.98.1:36451)
I20260928 09:03:08.666205 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40197
W20260928 09:03:08.747152 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:08.747424 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:08.747460 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:08.748766 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:08.748823 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:08.748862 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:03:08.750216 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:08.750487 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:08.750787 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:08.755879 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.756002 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.755880 19235 server_base.cc:1034] running on GCE node
W20260928 09:03:08.755973 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.756341 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:08.756529 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:08.757670 19235 hybrid_clock.cc:648] HybridClock initialized: now 1790586188757649 us; error 34 us; skew 500 ppm
I20260928 09:03:08.758973 19235 webserver.cc:460] Webserver started at http://127.0.98.2:42261/ using document root <none> and password file <none>
I20260928 09:03:08.759230 19235 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:08.759317 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:08.759477 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:08.760658 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "84f217dba96b43fa92b7a76c765dbfbe"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "8f4884304d970bc2c786325b3af1b224"
server_key_iv: "fb04272cfdf68e1f58af38b0ad8da53b"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.761057 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84f217dba96b43fa92b7a76c765dbfbe"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "8f4884304d970bc2c786325b3af1b224"
server_key_iv: "fb04272cfdf68e1f58af38b0ad8da53b"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.762434 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:08.763301 19249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.763516 19235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:08.763620 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "84f217dba96b43fa92b7a76c765dbfbe"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "8f4884304d970bc2c786325b3af1b224"
server_key_iv: "fb04272cfdf68e1f58af38b0ad8da53b"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.763700 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:08.772346 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:08.772709 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:08.772883 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:08.773128 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:08.773458 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:08.773512 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.773546 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:08.773573 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.779727 19235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35179
I20260928 09:03:08.779778 19362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35179 every 8 connection(s)
I20260928 09:03:08.780125 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:03:08.785261 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.780966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49039 (local address 127.0.98.62:37177)
0928 09:03:08.781032 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.781034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.781292 (+ 258us) server_negotiation.cc:407] Connection header received
0928 09:03:08.781756 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.781756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.781769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.781835 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.782486 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.782981 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.783981 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.784161 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.784213 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.785042 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.785043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.785043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:08.785048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.785062 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:08.785066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.785104 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.785175 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.785201 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 09:03:08.785522 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.780966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37177 (local address 127.0.98.2:49039)
0928 09:03:08.781152 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.781193 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.781677 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.781886 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.781889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.781897 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:08.782399 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.782403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.783087 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.783088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.783907 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.783909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.784231 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.784231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.784364 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.784908 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:08.784909 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:08.784938 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:08.784961 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:08.785110 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:08.785113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:08.785115 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.785162 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.785235 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260928 09:03:08.786230 19363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37177
I20260928 09:03:08.786339 19363 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:08.786522 19363 heartbeater.cc:511] Master 127.0.98.62:37177 requested a full tablet report, sending...
I20260928 09:03:08.786902 19044 ts_manager.cc:194] Registered new tserver with Master: 84f217dba96b43fa92b7a76c765dbfbe (127.0.98.2:35179)
I20260928 09:03:08.787238 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49039
I20260928 09:03:08.787839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19235
I20260928 09:03:08.787958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:03:08.788157 392 external_mini_cluster.cc:1428] Setting key a562ae1a67bd21e8edac187110db980e
I20260928 09:03:08.789053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37177
--builtin_ntp_servers=127.0.98.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:08.870702 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:08.870909 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:08.870931 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:08.871999 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:08.872033 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:08.872057 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:03:08.873410 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:08.873663 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:08.873987 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:08.879055 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.879209 19366 server_base.cc:1034] running on GCE node
W20260928 09:03:08.879200 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:08.879168 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:08.879805 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:08.880082 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:08.881255 19366 hybrid_clock.cc:648] HybridClock initialized: now 1790586188881236 us; error 30 us; skew 500 ppm
I20260928 09:03:08.882669 19366 webserver.cc:460] Webserver started at http://127.0.98.3:42383/ using document root <none> and password file <none>
I20260928 09:03:08.882932 19366 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:08.883005 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:08.883208 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:08.884320 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "0ff577e8ae674fd19d6310e515eea419"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d7ca4402de7c8c898d7d668a894ab907"
server_key_iv: "2b93f13eadde76a671b2a7fad16d0525"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.884756 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0ff577e8ae674fd19d6310e515eea419"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d7ca4402de7c8c898d7d668a894ab907"
server_key_iv: "2b93f13eadde76a671b2a7fad16d0525"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.886039 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:08.886783 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.886951 19366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:08.887048 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "0ff577e8ae674fd19d6310e515eea419"
format_stamp: "Formatted at 2026-09-28 09:03:08 on dist-test-slave-286n"
server_key: "d7ca4402de7c8c898d7d668a894ab907"
server_key_iv: "2b93f13eadde76a671b2a7fad16d0525"
server_key_version: "encryptionkey@0"
I20260928 09:03:08.887127 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:08.907469 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:08.907816 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:08.907980 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:08.908229 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:08.908566 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:08.908624 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.908686 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:08.908708 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:08.914758 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37149
I20260928 09:03:08.914801 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37149 every 8 connection(s)
I20260928 09:03:08.915145 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:03:08.920249 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.915864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34295 (local address 127.0.98.62:37177)
0928 09:03:08.915931 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.915933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.916212 (+ 279us) server_negotiation.cc:407] Connection header received
0928 09:03:08.916676 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.916678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.916687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.916740 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.917392 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.917901 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.918889 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.919100 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.919164 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.919964 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.919965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.919965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:08.919970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.919985 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:08.919989 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.920034 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.920117 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.920189 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 09:03:08.920393 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.915839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37177 (local address 127.0.98.3:34295)
0928 09:03:08.916034 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.916076 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.916564 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.916800 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.916803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.916811 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:08.917319 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.917324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.917990 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.917991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.918784 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.918786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.919184 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.919185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.919298 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.919821 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:08.919822 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:08.919849 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:08.919871 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:08.920046 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:08.920050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:08.920052 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.920103 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.920184 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 09:03:08.920962 19494 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37177
I20260928 09:03:08.921053 19494 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:08.921196 19494 heartbeater.cc:511] Master 127.0.98.62:37177 requested a full tablet report, sending...
I20260928 09:03:08.921571 19044 ts_manager.cc:194] Registered new tserver with Master: 0ff577e8ae674fd19d6310e515eea419 (127.0.98.3:37149)
I20260928 09:03:08.921904 19044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34295
I20260928 09:03:08.924391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19366
I20260928 09:03:08.924523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:03:08.924752 392 external_mini_cluster.cc:1428] Setting key fde06e28f456a6a3a7574ca0a360932d
I20260928 09:03:08.925784 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:03:08.930557 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.927163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.0.98.62:37177)
0928 09:03:08.927248 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.927250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.927419 (+ 169us) server_negotiation.cc:407] Connection header received
0928 09:03:08.927454 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.927455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.927466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.927535 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:08.927770 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.928394 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.929278 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.929505 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.929565 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.930281 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:08.930282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:08.930283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:08.930286 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:08.930301 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:08.930305 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:08.930381 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.930486 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.930496 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 09:03:08.932286 19044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:03:08.933841 19044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:03:08.940285 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.936079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37149 (local address 127.0.0.1:37050)
0928 09:03:08.936410 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.936438 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.936493 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.936946 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.936949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.936953 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.937030 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.937032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.937924 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.937925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.939085 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.939088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.940005 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.940006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.940082 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.940097 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.940165 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.940218 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":9,"thread_start_us":136,"threads_started":1}
I20260928 09:03:08.940513 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.936079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36451 (local address 127.0.0.1:53130)
0928 09:03:08.936397 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.936427 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.936479 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.937020 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.937021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.937023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.937069 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.937071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.938353 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.938354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.939331 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.939333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.940289 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.940290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.940350 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.940353 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.940403 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.940441 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
I20260928 09:03:08.940641 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.936180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.0.98.3:37149)
0928 09:03:08.936755 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.936758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.936767 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:03:08.936802 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.936805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.936819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.936885 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.937112 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.937801 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.939209 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.939915 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.939993 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.940083 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.940507 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.940549 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":87,"threads_started":1}
I20260928 09:03:08.940781 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.936587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.0.98.1:36451)
0928 09:03:08.936807 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.936812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.936822 (+ 10us) server_negotiation.cc:407] Connection header received
0928 09:03:08.936862 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.936864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.936880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.936955 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.937152 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.938254 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.939403 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.940114 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.940175 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.940254 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.940661 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.940703 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":110,"threads_started":1}
I20260928 09:03:08.941359 19166 tablet_service.cc:1467] Processing CreateTablet for tablet eb4d7258f83445a4a57253a521f8a0b9 (DEFAULT_TABLE table=test-table [id=56707e4bdca94110ba727d1226897fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:08.941359 19428 tablet_service.cc:1467] Processing CreateTablet for tablet eb4d7258f83445a4a57253a521f8a0b9 (DEFAULT_TABLE table=test-table [id=56707e4bdca94110ba727d1226897fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:08.941728 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4d7258f83445a4a57253a521f8a0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.941728 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4d7258f83445a4a57253a521f8a0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.945083 19508 tablet_bootstrap.cc:492] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: Bootstrap starting.
I20260928 09:03:08.945288 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.936393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.0.98.2:35179)
0928 09:03:08.936577 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:08.936581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:08.936686 (+ 105us) server_negotiation.cc:407] Connection header received
0928 09:03:08.936759 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:08.936761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:08.936780 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:08.936861 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:08.937094 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.938245 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.940159 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.940987 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.941116 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.941278 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:08.945153 (+ 3875us) server_negotiation.cc:299] Negotiation successful
0928 09:03:08.945202 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260928 09:03:08.945513 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:08.936088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35179 (local address 127.0.0.1:57424)
0928 09:03:08.936591 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:08.936600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:08.936687 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:08.936945 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:08.936948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:08.936950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:08.937030 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.937033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.939397 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.939397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.940078 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:08.940079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:08.944872 (+ 4793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:08.944873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:08.944922 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:08.944946 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 09:03:08.945007 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 09:03:08.945413 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":5,"thread_start_us":100,"threads_started":1}
I20260928 09:03:08.945917 19508 tablet_bootstrap.cc:654] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.946228 19508 log.cc:824] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.946264 19297 tablet_service.cc:1467] Processing CreateTablet for tablet eb4d7258f83445a4a57253a521f8a0b9 (DEFAULT_TABLE table=test-table [id=56707e4bdca94110ba727d1226897fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:08.946573 19509 tablet_bootstrap.cc:492] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Bootstrap starting.
I20260928 09:03:08.946631 19297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4d7258f83445a4a57253a521f8a0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:08.947132 19509 tablet_bootstrap.cc:654] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.947515 19509 log.cc:824] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.948051 19508 tablet_bootstrap.cc:492] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: No bootstrap required, opened a new log
I20260928 09:03:08.948144 19508 ts_tablet_manager.cc:1397] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:03:08.949003 19509 tablet_bootstrap.cc:492] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: No bootstrap required, opened a new log
I20260928 09:03:08.949072 19509 ts_tablet_manager.cc:1397] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 09:03:08.949581 19512 tablet_bootstrap.cc:492] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Bootstrap starting.
I20260928 09:03:08.950248 19512 tablet_bootstrap.cc:654] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:08.950595 19512 log.cc:824] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:08.950524 19508 raft_consensus.cc:348] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:08.950697 19508 raft_consensus.cc:374] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.950737 19508 raft_consensus.cc:729] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013c6c72256340739e034898b13a82a9, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.950856 19508 consensus_queue.cc:260] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:08.951128 19508 ts_tablet_manager.cc:1428] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:03:08.951184 19232 heartbeater.cc:503] Master 127.0.98.62:37177 was elected leader, sending a full tablet report...
I20260928 09:03:08.951185 19509 raft_consensus.cc:348] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:08.951313 19512 tablet_bootstrap.cc:492] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: No bootstrap required, opened a new log
I20260928 09:03:08.951325 19509 raft_consensus.cc:374] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.951354 19512 ts_tablet_manager.cc:1397] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:03:08.951357 19509 raft_consensus.cc:729] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff577e8ae674fd19d6310e515eea419, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.951471 19509 consensus_queue.cc:260] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:08.951851 19509 ts_tablet_manager.cc:1428] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:03:08.951977 19494 heartbeater.cc:503] Master 127.0.98.62:37177 was elected leader, sending a full tablet report...
I20260928 09:03:08.953557 19512 raft_consensus.cc:348] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:08.953687 19512 raft_consensus.cc:374] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:08.953709 19512 raft_consensus.cc:729] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f217dba96b43fa92b7a76c765dbfbe, State: Initialized, Role: FOLLOWER
I20260928 09:03:08.953781 19512 consensus_queue.cc:260] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:08.953945 19512 ts_tablet_manager.cc:1428] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:03:08.954026 19363 heartbeater.cc:503] Master 127.0.98.62:37177 was elected leader, sending a full tablet report...
W20260928 09:03:09.030738 19364 tablet.cc:2369] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:03:09.156481 19233 tablet.cc:2369] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 09:03:09.165715 19495 tablet.cc:2369] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:09.329710 19515 raft_consensus.cc:482] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:03:09.329834 19515 raft_consensus.cc:504] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:09.330188 19515 leader_election.cc:288] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84f217dba96b43fa92b7a76c765dbfbe (127.0.98.2:35179), 013c6c72256340739e034898b13a82a9 (127.0.98.1:36451)
I20260928 09:03:09.334244 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:09.330443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45905 (local address 127.0.98.2:35179)
0928 09:03:09.330504 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:09.330506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:09.330518 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:03:09.330604 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:09.330605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:09.330614 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:09.330710 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:09.330912 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.331664 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.333272 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.333833 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.333913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:09.333924 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:09.334123 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 09:03:09.334170 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:03:09.334244 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:09.330531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34715 (local address 127.0.98.1:36451)
0928 09:03:09.330583 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:09.330585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:09.330670 (+ 85us) server_negotiation.cc:407] Connection header received
0928 09:03:09.330698 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:09.330699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:09.330710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:09.330767 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:09.330911 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.331679 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.333273 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.333833 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.333913 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:09.333925 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:09.334133 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 09:03:09.334171 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:09.334239 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:09.330387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35179 (local address 127.0.98.3:45905)
0928 09:03:09.330451 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:09.330463 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:09.330521 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:09.330731 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:09.330732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:09.330734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:09.330814 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:09.330816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.331773 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.331774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:09.333159 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:09.333162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.333936 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.333941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:09.334018 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:09.334022 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:09.334086 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 09:03:09.334136 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":78}
I20260928 09:03:09.334245 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:09.330404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36451 (local address 127.0.98.3:34715)
0928 09:03:09.330589 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:09.330599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:09.330654 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:09.330772 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:09.330773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:09.330774 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:09.330837 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:09.330838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.331773 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.331774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:09.333159 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:09.333162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.333953 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.333954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:09.334053 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:09.334057 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:09.334109 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 09:03:09.334136 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":60,"thread_start_us":104,"threads_started":1}
I20260928 09:03:09.334643 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d7258f83445a4a57253a521f8a0b9" candidate_uuid: "0ff577e8ae674fd19d6310e515eea419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013c6c72256340739e034898b13a82a9" is_pre_election: true
I20260928 09:03:09.334653 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d7258f83445a4a57253a521f8a0b9" candidate_uuid: "0ff577e8ae674fd19d6310e515eea419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f217dba96b43fa92b7a76c765dbfbe" is_pre_election: true
I20260928 09:03:09.334800 19317 raft_consensus.cc:2454] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff577e8ae674fd19d6310e515eea419 in term 0.
I20260928 09:03:09.334815 19186 raft_consensus.cc:2454] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff577e8ae674fd19d6310e515eea419 in term 0.
I20260928 09:03:09.335079 19382 leader_election.cc:302] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff577e8ae674fd19d6310e515eea419, 84f217dba96b43fa92b7a76c765dbfbe; no voters:
I20260928 09:03:09.335297 19515 raft_consensus.cc:2781] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:03:09.335340 19515 raft_consensus.cc:482] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:03:09.335361 19515 raft_consensus.cc:3037] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:09.336086 19515 raft_consensus.cc:504] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:09.336236 19515 leader_election.cc:288] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [CANDIDATE]: Term 1 election: Requested vote from peers 84f217dba96b43fa92b7a76c765dbfbe (127.0.98.2:35179), 013c6c72256340739e034898b13a82a9 (127.0.98.1:36451)
I20260928 09:03:09.336390 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d7258f83445a4a57253a521f8a0b9" candidate_uuid: "0ff577e8ae674fd19d6310e515eea419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f217dba96b43fa92b7a76c765dbfbe"
I20260928 09:03:09.336390 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d7258f83445a4a57253a521f8a0b9" candidate_uuid: "0ff577e8ae674fd19d6310e515eea419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013c6c72256340739e034898b13a82a9"
I20260928 09:03:09.336478 19186 raft_consensus.cc:3037] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:09.336495 19317 raft_consensus.cc:3037] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:09.337397 19186 raft_consensus.cc:2454] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff577e8ae674fd19d6310e515eea419 in term 1.
I20260928 09:03:09.337397 19317 raft_consensus.cc:2454] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff577e8ae674fd19d6310e515eea419 in term 1.
I20260928 09:03:09.337622 19383 leader_election.cc:302] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013c6c72256340739e034898b13a82a9, 0ff577e8ae674fd19d6310e515eea419; no voters:
I20260928 09:03:09.337734 19515 raft_consensus.cc:2781] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:09.337894 19515 raft_consensus.cc:686] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff577e8ae674fd19d6310e515eea419, State: Running, Role: LEADER
I20260928 09:03:09.338049 19515 consensus_queue.cc:237] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } }
I20260928 09:03:09.338851 19044 catalog_manager.cc:5495] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ff577e8ae674fd19d6310e515eea419 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0ff577e8ae674fd19d6310e515eea419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff577e8ae674fd19d6310e515eea419" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 } health_report { overall_health: UNKNOWN } } }
I20260928 09:03:09.381497 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:09.379002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.98.3:37149)
0928 09:03:09.379047 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:09.379049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:09.379074 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:03:09.379144 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:09.379145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:09.379167 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:09.379222 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:09.379423 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.379876 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.380608 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:09.380990 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:09.381044 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:09.381200 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:09.381292 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:09.381363 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:09.381407 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 09:03:09.381415 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:03:09.383513 19317 raft_consensus.cc:1261] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 1 FOLLOWER]: Refusing update from remote peer 0ff577e8ae674fd19d6310e515eea419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:09.383513 19186 raft_consensus.cc:1261] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 1 FOLLOWER]: Refusing update from remote peer 0ff577e8ae674fd19d6310e515eea419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:09.383845 19515 consensus_queue.cc:1035] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "013c6c72256340739e034898b13a82a9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:09.384025 19518 consensus_queue.cc:1035] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f217dba96b43fa92b7a76c765dbfbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:09.388613 19044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:09.388746 19044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:09.389539 19044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0e23039bf3740b5a24f62f1c023ad57: Sending DeleteTablet for 3 replicas of tablet eb4d7258f83445a4a57253a521f8a0b9
I20260928 09:03:09.389848 19428 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4d7258f83445a4a57253a521f8a0b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:09 UTC) from {username='slave'} at 127.0.0.1:37050
I20260928 09:03:09.389878 19297 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4d7258f83445a4a57253a521f8a0b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:09 UTC) from {username='slave'} at 127.0.0.1:57424
I20260928 09:03:09.389873 19166 tablet_service.cc:1514] Processing DeleteTablet for tablet eb4d7258f83445a4a57253a521f8a0b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:09 UTC) from {username='slave'} at 127.0.0.1:53130
I20260928 09:03:09.390115 19531 tablet_replica.cc:330] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: stopping tablet replica
I20260928 09:03:09.390172 19532 tablet_replica.cc:330] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9: stopping tablet replica
I20260928 09:03:09.390280 19532 raft_consensus.cc:2229] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:09.390286 19531 raft_consensus.cc:2229] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:09.390390 19532 raft_consensus.cc:2258] T eb4d7258f83445a4a57253a521f8a0b9 P 013c6c72256340739e034898b13a82a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:09.390470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19104
I20260928 09:03:09.391952 19531 raft_consensus.cc:2258] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:09.392498 19531 ts_tablet_manager.cc:1905] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:09.394771 19531 ts_tablet_manager.cc:1918] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:09.394891 19531 log.cc:1196] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/eb4d7258f83445a4a57253a521f8a0b9
I20260928 09:03:09.395215 19531 ts_tablet_manager.cc:1939] T eb4d7258f83445a4a57253a521f8a0b9 P 84f217dba96b43fa92b7a76c765dbfbe: Deleting consensus metadata
I20260928 09:03:09.395723 19029 catalog_manager.cc:4841] TS 84f217dba96b43fa92b7a76c765dbfbe (127.0.98.2:35179): tablet eb4d7258f83445a4a57253a521f8a0b9 (table test-table [id=56707e4bdca94110ba727d1226897fae]) successfully deleted
W20260928 09:03:09.397208 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36451 (error 108)
I20260928 09:03:09.397275 19530 tablet_replica.cc:330] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: stopping tablet replica
I20260928 09:03:09.397388 19530 raft_consensus.cc:2229] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:03:09.397440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19235
I20260928 09:03:09.397483 19530 raft_consensus.cc:2258] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:09.398516 19530 ts_tablet_manager.cc:1905] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:09.399808 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:09.399659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36451 (local address 127.0.0.1:53138)
0928 09:03:09.399723 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:09.399767 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 09:03:09.399992 19030 catalog_manager.cc:4584] TS 013c6c72256340739e034898b13a82a9 (127.0.98.1:36451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb4d7258f83445a4a57253a521f8a0b9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36451: connect: Connection refused (error 111)
I20260928 09:03:09.400036 19530 ts_tablet_manager.cc:1918] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:09.400095 19530 log.cc:1196] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/eb4d7258f83445a4a57253a521f8a0b9
I20260928 09:03:09.400345 19530 ts_tablet_manager.cc:1939] T eb4d7258f83445a4a57253a521f8a0b9 P 0ff577e8ae674fd19d6310e515eea419: Deleting consensus metadata
I20260928 09:03:09.400750 19028 catalog_manager.cc:4841] TS 0ff577e8ae674fd19d6310e515eea419 (127.0.98.3:37149): tablet eb4d7258f83445a4a57253a521f8a0b9 (table test-table [id=56707e4bdca94110ba727d1226897fae]) successfully deleted
I20260928 09:03:09.403033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19366
I20260928 09:03:09.408164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19013
2026-09-28T09:03:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:09 dist-test-slave-286n krb5kdc[19536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:09 dist-test-slave-286n krb5kdc[19536](info): set up 2 sockets
Sep 28 09:03:09 dist-test-slave-286n krb5kdc[19536](info): commencing operation
krb5kdc: starting...
W20260928 09:03:10.444537 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:10 dist-test-slave-286n krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:10.563416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44435
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:10.644166 19549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:10.644419 19549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:10.644456 19549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:10.645638 19549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:10.645715 19549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:10.645741 19549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:10.645761 19549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:10.645789 19549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:10.647568 19549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:10.647868 19549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:10.648111 19549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:10.652909 19557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:10.652910 19555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:10.652933 19554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:10.653266 19549 server_base.cc:1034] running on GCE node
I20260928 09:03:10.653648 19549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:10.653965 19549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:10.655119 19549 hybrid_clock.cc:648] HybridClock initialized: now 1790586190655109 us; error 33 us; skew 500 ppm
Sep 28 09:03:10 dist-test-slave-286n krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:10.664283 19549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:10.664906 19549 webserver.cc:460] Webserver started at http://127.0.98.62:39729/ using document root <none> and password file <none>
I20260928 09:03:10.665174 19549 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:10.665221 19549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:10.665385 19549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:10.666471 19549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "a9a4f44464bc4cf28cc91424bcdc34a1"
format_stamp: "Formatted at 2026-09-28 09:03:10 on dist-test-slave-286n"
server_key: "03157efa1433a31fa82cb4a9e1464e98"
server_key_iv: "ed93443c7540524807821d7364553300"
server_key_version: "encryptionkey@0"
I20260928 09:03:10.666834 19549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9a4f44464bc4cf28cc91424bcdc34a1"
format_stamp: "Formatted at 2026-09-28 09:03:10 on dist-test-slave-286n"
server_key: "03157efa1433a31fa82cb4a9e1464e98"
server_key_iv: "ed93443c7540524807821d7364553300"
server_key_version: "encryptionkey@0"
I20260928 09:03:10.668195 19549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:10.669003 19564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:10.669260 19549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 09:03:10.669369 19549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "a9a4f44464bc4cf28cc91424bcdc34a1"
format_stamp: "Formatted at 2026-09-28 09:03:10 on dist-test-slave-286n"
server_key: "03157efa1433a31fa82cb4a9e1464e98"
server_key_iv: "ed93443c7540524807821d7364553300"
server_key_version: "encryptionkey@0"
I20260928 09:03:10.669456 19549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:10.687001 19549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:10.689050 19549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:10.689204 19549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:10.693357 19549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44435
I20260928 09:03:10.693418 19616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44435 every 8 connection(s)
I20260928 09:03:10.693794 19549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:10.694674 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:10.696965 19617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1: Bootstrap starting.
I20260928 09:03:10.697566 19617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:10.697856 19617 log.cc:824] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:10.698379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19549
I20260928 09:03:10.698570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:10.698755 392 external_mini_cluster.cc:1428] Setting key 293f54d03e19893582069e83cb6c64b2
I20260928 09:03:10.698763 19617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1: No bootstrap required, opened a new log
I20260928 09:03:10.700742 19617 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44435 } }
I20260928 09:03:10.700903 19617 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:10.700924 19617 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a4f44464bc4cf28cc91424bcdc34a1, State: Initialized, Role: FOLLOWER
I20260928 09:03:10.701097 19617 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44435 } }
I20260928 09:03:10.701161 19617 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:10.701184 19617 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:10.701232 19617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:10.701815 19617 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44435 } }
I20260928 09:03:10.701903 19617 leader_election.cc:302] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a4f44464bc4cf28cc91424bcdc34a1; no voters:
I20260928 09:03:10.702109 19617 leader_election.cc:288] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:10.702131 19622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:10.702256 19622 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [term 1 LEADER]: Becoming Leader. State: Replica: a9a4f44464bc4cf28cc91424bcdc34a1, State: Running, Role: LEADER
I20260928 09:03:10.702337 19622 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44435 } }
I20260928 09:03:10.702370 19617 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:10.702688 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a4f44464bc4cf28cc91424bcdc34a1. Latest consensus state: current_term: 1 leader_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44435 } } }
I20260928 09:03:10.702769 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:10.702694 19624 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a4f44464bc4cf28cc91424bcdc34a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44435 } } }
I20260928 09:03:10.702955 19624 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:10.703193 19628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:10.703677 19628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:10.706168 19628 catalog_manager.cc:1347] Generated new cluster ID: 773916cc4e174bac86df9a53c14e4e3c
I20260928 09:03:10.706225 19628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:10 dist-test-slave-286n krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:10.710681 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:10.699288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.0.98.62:44435)
0928 09:03:10.699497 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:10.699506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:10.699543 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:03:10.700021 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:10.700028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:10.700076 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:10.700230 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:10.700992 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:10.702481 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:10.703599 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:10.704294 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:10.704421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:10.707129 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:10.707133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:10.707134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:10.707146 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:10.708915 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:10.709319 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:10.709322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:10.709323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:10.709412 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:10.709718 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:10.709719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:10.709719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:10.709980 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:10.710150 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:10.710253 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 09:03:10.710395 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":105,"threads_started":1}
I20260928 09:03:10.715891 19628 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:10.716596 19628 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:10.731786 19628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1: Generated new TSK 0
I20260928 09:03:10.732019 19628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 09:03:10.759256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44435
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:10.846464 19643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:10.846700 19643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:10.846740 19643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:10.847904 19643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:10.847973 19643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:10.848021 19643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:03:10.849577 19643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:10.849848 19643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:10.850067 19643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:10.854985 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:10.855000 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:10.855043 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:10.855336 19643 server_base.cc:1034] running on GCE node
I20260928 09:03:10.855548 19643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:10.855789 19643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:10.856948 19643 hybrid_clock.cc:648] HybridClock initialized: now 1790586190856994 us; error 87 us; skew 500 ppm
Sep 28 09:03:10 dist-test-slave-286n krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:10.859404 19643 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 09:03:10.859848 19643 webserver.cc:460] Webserver started at http://127.0.98.1:44437/ using document root <none> and password file <none>
I20260928 09:03:10.860076 19643 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:10.860122 19643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:10.860270 19643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:10.861356 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1860f0c0ed24939945786d64f0afeff"
format_stamp: "Formatted at 2026-09-28 09:03:10 on dist-test-slave-286n"
server_key: "d8ea836bc1e5fe27f184af78a787fad1"
server_key_iv: "f3f39114cfe52cd2e3de6dc700b1d914"
server_key_version: "encryptionkey@0"
I20260928 09:03:10.861706 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1860f0c0ed24939945786d64f0afeff"
format_stamp: "Formatted at 2026-09-28 09:03:10 on dist-test-slave-286n"
server_key: "d8ea836bc1e5fe27f184af78a787fad1"
server_key_iv: "f3f39114cfe52cd2e3de6dc700b1d914"
server_key_version: "encryptionkey@0"
I20260928 09:03:10.863034 19643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:10.863771 19658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:10.863941 19643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:10.864040 19643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "e1860f0c0ed24939945786d64f0afeff"
format_stamp: "Formatted at 2026-09-28 09:03:10 on dist-test-slave-286n"
server_key: "d8ea836bc1e5fe27f184af78a787fad1"
server_key_iv: "f3f39114cfe52cd2e3de6dc700b1d914"
server_key_version: "encryptionkey@0"
I20260928 09:03:10.864111 19643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:10.884435 19643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:10.886852 19643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:10.887018 19643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:10.887267 19643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:10.887583 19643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:10.887635 19643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:10.887677 19643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:10.887705 19643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:10.893792 19643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40235
I20260928 09:03:10.893836 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40235 every 8 connection(s)
I20260928 09:03:10.894181 19643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:03:10.894960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19643
I20260928 09:03:10.895092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:03:10.895336 392 external_mini_cluster.cc:1428] Setting key f2c0a941ebcfd40ddbae85528dadd0fb
Sep 28 09:03:10 dist-test-slave-286n krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 09:03:10.905663 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:10.894992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40145 (local address 127.0.98.62:44435)
0928 09:03:10.895069 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:10.895072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:10.895320 (+ 248us) server_negotiation.cc:407] Connection header received
0928 09:03:10.896507 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:10.896508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:10.896517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:10.896558 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:10.897474 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:10.898332 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:10.899441 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:10.899840 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:10.899923 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:10.902621 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:10.902623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:10.902624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:10.902633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:10.904369 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:10.904860 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:10.904861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:10.904862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:10.904912 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:10.905134 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:10.905135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:10.905135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:10.905228 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:10.905272 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:10.905526 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 09:03:10.905595 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 09:03:10.905922 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:10.894912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44435 (local address 127.0.98.1:40145)
0928 09:03:10.895176 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:10.895231 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:10.896403 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:10.896655 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:10.896660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:10.897084 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:10.897371 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:10.897378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:10.898427 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:10.898428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:10.899352 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:10.899355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:10.899946 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:10.899947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:10.900059 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:10.900675 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:10.900686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:10.902394 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:10.904502 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:10.904509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:10.904510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:10.904803 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:10.904974 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:10.904975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:10.904975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:10.905091 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:10.905277 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:10.905281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:10.905398 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 09:03:10.905504 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 09:03:10.905663 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260928 09:03:10.906474 19772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44435
I20260928 09:03:10.906592 19772 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:10.906771 19772 heartbeater.cc:511] Master 127.0.98.62:44435 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 09:03:10.907456 19581 ts_manager.cc:194] Registered new tserver with Master: e1860f0c0ed24939945786d64f0afeff (127.0.98.1:40235)
I20260928 09:03:10.908438 19581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 09:03:10.923722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44435
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:11.003780 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:11.004009 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:11.004048 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:11.005265 19777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:11.005330 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:11.005374 19777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:03:11.006841 19777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:11.007113 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:11.007349 19777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:11.012439 19785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.012498 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.012430 19782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:11.012713 19777 server_base.cc:1034] running on GCE node
I20260928 09:03:11.012977 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:11.013226 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:11.014393 19777 hybrid_clock.cc:648] HybridClock initialized: now 1790586191014374 us; error 35 us; skew 500 ppm
Sep 28 09:03:11 dist-test-slave-286n krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:11.016898 19777 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 09:03:11.017364 19777 webserver.cc:460] Webserver started at http://127.0.98.2:33857/ using document root <none> and password file <none>
I20260928 09:03:11.017647 19777 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:11.017722 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:11.017907 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:11.018973 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "c6d2cca9ae483c6b202c535c6a69a9f1"
server_key_iv: "0378e11ab1865d789ba434a2ff45e399"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.019348 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "c6d2cca9ae483c6b202c535c6a69a9f1"
server_key_iv: "0378e11ab1865d789ba434a2ff45e399"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.020725 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 09:03:11.021528 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.021697 19777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:11.021795 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "c6d2cca9ae483c6b202c535c6a69a9f1"
server_key_iv: "0378e11ab1865d789ba434a2ff45e399"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.021888 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:11.050238 19777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:11.052816 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:11.052975 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:11.053237 19777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:11.053555 19777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:11.053609 19777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.053651 19777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:11.053678 19777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.059588 19777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36009
I20260928 09:03:11.059633 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36009 every 8 connection(s)
I20260928 09:03:11.059962 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 09:03:11 dist-test-slave-286n krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:11.068840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19777
I20260928 09:03:11.068962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:03:11.069192 392 external_mini_cluster.cc:1428] Setting key ecf8e683846216410a067976404383db
I20260928 09:03:11.069917 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.060734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54751 (local address 127.0.98.62:44435)
0928 09:03:11.060780 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.060782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.061058 (+ 276us) server_negotiation.cc:407] Connection header received
0928 09:03:11.062042 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.062043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.062055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.062113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.062760 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.063275 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.064301 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.064500 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.064586 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.066912 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.066914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.066914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:11.066923 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.068297 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:11.068742 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:11.068743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:11.068744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:11.068805 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:11.069112 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:11.069114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:11.069115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:11.069281 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.069363 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.069723 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.069817 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:11.070103 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.060705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44435 (local address 127.0.98.2:54751)
0928 09:03:11.060899 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.060964 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.061924 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.062186 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.062190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.062458 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:11.062652 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.062656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.063373 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.063374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.064206 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.064208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.064596 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.064597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.064765 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.065347 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:11.065356 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:11.066811 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:11.068387 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:11.068394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:11.068397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:11.068621 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:11.068876 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:11.068877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:11.068878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:11.069030 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:11.069352 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:11.069361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:11.069559 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.069666 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.069840 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260928 09:03:11.070653 19906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44435
I20260928 09:03:11.070756 19906 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:11.070951 19906 heartbeater.cc:511] Master 127.0.98.62:44435 requested a full tablet report, sending...
I20260928 09:03:11.071367 19581 ts_manager.cc:194] Registered new tserver with Master: b9d2cec0475a49bb83e7b27ea47f0e8f (127.0.98.2:36009)
I20260928 09:03:11.071910 19581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54751
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 09:03:11.095383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44435
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:11.177038 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:11.177246 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:11.177271 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:11.178464 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:11.178510 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:11.178570 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:03:11.180114 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:11.180385 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:11.180666 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:11.185693 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.185765 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.185693 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:11.185866 19911 server_base.cc:1034] running on GCE node
I20260928 09:03:11.186153 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:11.186424 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:11.187597 19911 hybrid_clock.cc:648] HybridClock initialized: now 1790586191187586 us; error 32 us; skew 500 ppm
Sep 28 09:03:11 dist-test-slave-286n krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:11.190208 19911 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 09:03:11.190688 19911 webserver.cc:460] Webserver started at http://127.0.98.3:44197/ using document root <none> and password file <none>
I20260928 09:03:11.190929 19911 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:11.190975 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:11.191156 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:11.192226 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "cab3b9cd024d47898e73308ba197e575"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "308ce49b9c0833521dee332296e50e1b"
server_key_iv: "17662c6ca0045c75e10747ba7a80e246"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.192579 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cab3b9cd024d47898e73308ba197e575"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "308ce49b9c0833521dee332296e50e1b"
server_key_iv: "17662c6ca0045c75e10747ba7a80e246"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.193938 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:11.194769 19926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.194964 19911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:11.195065 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "cab3b9cd024d47898e73308ba197e575"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "308ce49b9c0833521dee332296e50e1b"
server_key_iv: "17662c6ca0045c75e10747ba7a80e246"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.195170 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:11.206202 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:11.208097 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:11.208274 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:11.208525 19911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:11.208963 19911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:11.209018 19911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.209061 19911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:11.209095 19911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.215044 19911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45789
I20260928 09:03:11.215086 20039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45789 every 8 connection(s)
I20260928 09:03:11.215445 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:03:11.220487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 19911
I20260928 09:03:11.220590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:03:11.220799 392 external_mini_cluster.cc:1428] Setting key 1aa6ceb1b622197837c41908bccf2431
Sep 28 09:03:11 dist-test-slave-286n krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:11.225391 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.216205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40177 (local address 127.0.98.62:44435)
0928 09:03:11.216269 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.216272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.216468 (+ 196us) server_negotiation.cc:407] Connection header received
0928 09:03:11.217534 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.217536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.217548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.217594 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.218284 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.218800 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.219819 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.220029 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.220124 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.222678 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.222679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.222679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:11.222686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.224091 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:11.224436 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:11.224437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:11.224438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:11.224509 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:11.224861 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:11.224862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:11.224862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:11.224970 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.225026 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.225265 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.225327 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:03:11.225586 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.216192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44435 (local address 127.0.98.3:40177)
0928 09:03:11.216347 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.216382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.217420 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.217705 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.217708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.217968 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:11.218175 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.218180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.218894 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.218895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.219722 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.219724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.220135 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.220135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.220260 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.220941 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:11.220951 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:11.222592 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:11.224162 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:11.224167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:11.224168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:11.224372 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:11.224690 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:11.224691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:11.224692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:11.224801 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:11.225030 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:11.225035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:11.225150 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.225226 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.225344 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 09:03:11.226240 20040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44435
I20260928 09:03:11.226351 20040 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:11.226532 20040 heartbeater.cc:511] Master 127.0.98.62:44435 requested a full tablet report, sending...
I20260928 09:03:11.226935 19581 ts_manager.cc:194] Registered new tserver with Master: cab3b9cd024d47898e73308ba197e575 (127.0.98.3:45789)
I20260928 09:03:11.227319 19581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40177
I20260928 09:03:11.232602 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:03:11.239763 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.233985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.0.98.62:44435)
0928 09:03:11.234066 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.234068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.234104 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:03:11.234160 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.234161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.234169 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.234221 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.234753 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.235239 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.235937 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.236126 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.236181 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.237367 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.237368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.237368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:11.237374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.238779 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:11.239099 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:11.239100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:11.239100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:11.239157 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:11.239325 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:11.239326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:11.239326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:11.239467 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.239545 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.239637 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.239698 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 09:03:11.241348 19581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:03:11.242933 19581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:03:11.249775 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.245462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.98.1:40235)
0928 09:03:11.245605 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.245611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.245984 (+ 373us) server_negotiation.cc:407] Connection header received
0928 09:03:11.246034 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.246036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.246051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.246135 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.246391 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.247437 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.248616 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.249353 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.249434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.249498 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.249636 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.249688 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260928 09:03:11.249918 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.245359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40235 (local address 127.0.0.1:51808)
0928 09:03:11.245840 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.245852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.245914 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.246257 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.246258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.246260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.246312 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.246313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.247539 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.247540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.248524 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.248527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.249458 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.249459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.249522 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.249532 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.249603 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.249655 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260928 09:03:11.250141 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.245357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36009 (local address 127.0.0.1:43228)
0928 09:03:11.245515 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.245557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.245638 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.246102 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.246105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.246107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.246177 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.246179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.247515 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.247516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.248786 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.248787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.249712 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.249713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.249788 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.249792 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.249949 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.250072 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":30,"thread_start_us":46,"threads_started":1}
I20260928 09:03:11.250404 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.245638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.0.98.2:36009)
0928 09:03:11.245826 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.245830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.245838 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:11.245897 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.245900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.245917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.245991 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.246260 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.247409 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.248868 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.249617 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.249690 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.249747 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.250275 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.250333 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260928 09:03:11.250666 19706 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a44230c8b444dbc4e6f13b6d71439 (DEFAULT_TABLE table=test-table [id=87ec49dfdfcb4c5fa94d96548a42faae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:11.250945 19706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a44230c8b444dbc4e6f13b6d71439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.253361 19840 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a44230c8b444dbc4e6f13b6d71439 (DEFAULT_TABLE table=test-table [id=87ec49dfdfcb4c5fa94d96548a42faae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:11.253718 19840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a44230c8b444dbc4e6f13b6d71439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.253790 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.245547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45789 (local address 127.0.0.1:34688)
0928 09:03:11.246285 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.246292 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.246351 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.246521 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.246521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.246523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.246563 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.246564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.247379 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.247381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.248779 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.248783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.253473 (+ 4690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.253474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.253585 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.253590 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.253674 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.253711 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":82,"thread_start_us":83,"threads_started":1}
I20260928 09:03:11.253886 20057 tablet_bootstrap.cc:492] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: Bootstrap starting.
I20260928 09:03:11.254041 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.245782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.98.3:45789)
0928 09:03:11.245997 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.246002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.246352 (+ 350us) server_negotiation.cc:407] Connection header received
0928 09:03:11.246420 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.246422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.246443 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.246524 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.246615 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.247277 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.249787 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.253355 (+ 3568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.253468 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.253530 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.253900 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.253953 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260928 09:03:11.254552 20057 tablet_bootstrap.cc:654] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.255134 20057 log.cc:824] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.255980 20057 tablet_bootstrap.cc:492] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: No bootstrap required, opened a new log
I20260928 09:03:11.256071 20057 ts_tablet_manager.cc:1397] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:11.256217 19974 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a44230c8b444dbc4e6f13b6d71439 (DEFAULT_TABLE table=test-table [id=87ec49dfdfcb4c5fa94d96548a42faae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:11.256532 19974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a44230c8b444dbc4e6f13b6d71439. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.257243 20059 tablet_bootstrap.cc:492] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Bootstrap starting.
I20260928 09:03:11.257985 20059 tablet_bootstrap.cc:654] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.258188 20057 raft_consensus.cc:348] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.258430 20057 raft_consensus.cc:374] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.258474 20057 raft_consensus.cc:729] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1860f0c0ed24939945786d64f0afeff, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.258605 20057 consensus_queue.cc:260] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.258620 20059 log.cc:824] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.258914 20057 ts_tablet_manager.cc:1428] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:03:11.258916 19772 heartbeater.cc:503] Master 127.0.98.62:44435 was elected leader, sending a full tablet report...
I20260928 09:03:11.259511 20062 tablet_bootstrap.cc:492] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Bootstrap starting.
I20260928 09:03:11.260195 20062 tablet_bootstrap.cc:654] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.260262 20059 tablet_bootstrap.cc:492] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: No bootstrap required, opened a new log
I20260928 09:03:11.260339 20059 ts_tablet_manager.cc:1397] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 09:03:11.260908 20062 log.cc:824] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.261678 20062 tablet_bootstrap.cc:492] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: No bootstrap required, opened a new log
I20260928 09:03:11.261750 20062 ts_tablet_manager.cc:1397] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 09:03:11.262651 20059 raft_consensus.cc:348] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.262862 20059 raft_consensus.cc:374] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.262904 20059 raft_consensus.cc:729] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d2cec0475a49bb83e7b27ea47f0e8f, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.263022 20059 consensus_queue.cc:260] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.263340 20059 ts_tablet_manager.cc:1428] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 09:03:11.263353 19906 heartbeater.cc:503] Master 127.0.98.62:44435 was elected leader, sending a full tablet report...
I20260928 09:03:11.264035 20062 raft_consensus.cc:348] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.264230 20062 raft_consensus.cc:374] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.264281 20062 raft_consensus.cc:729] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab3b9cd024d47898e73308ba197e575, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.264413 20062 consensus_queue.cc:260] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.264777 20040 heartbeater.cc:503] Master 127.0.98.62:44435 was elected leader, sending a full tablet report...
I20260928 09:03:11.264783 20062 ts_tablet_manager.cc:1428] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 09:03:11.310596 19907 tablet.cc:2369] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:11.330710 20064 raft_consensus.cc:482] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:03:11.330850 20064 raft_consensus.cc:504] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.331319 20064 leader_election.cc:288] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1860f0c0ed24939945786d64f0afeff (127.0.98.1:40235), cab3b9cd024d47898e73308ba197e575 (127.0.98.3:45789)
I20260928 09:03:11.334470 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.331585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48645 (local address 127.0.98.1:40235)
0928 09:03:11.331673 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.331675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.331687 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:03:11.331719 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.331720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.331735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.331820 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.332011 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.332516 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.333608 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.334073 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.334162 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.334175 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.334350 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.334389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 09:03:11.334470 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.331516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40235 (local address 127.0.98.2:48645)
0928 09:03:11.331580 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.331592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.331645 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.331839 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.331840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.331841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.331930 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.331932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.332618 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.332619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.333500 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.333502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.334178 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.334179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.334266 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.334270 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.334344 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.334387 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260928 09:03:11.334535 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.331594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41811 (local address 127.0.98.3:45789)
0928 09:03:11.331667 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.331669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.331839 (+ 170us) server_negotiation.cc:407] Connection header received
0928 09:03:11.331890 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.331891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.331902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.332036 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.332213 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.332800 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.333720 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.334182 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.334264 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.334274 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.334419 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.334456 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:03:11.334769 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.331531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45789 (local address 127.0.98.2:41811)
0928 09:03:11.331744 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.331753 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.331833 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.332043 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.332044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.332046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.332119 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.332121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.332868 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.332869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.333629 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.333631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.334267 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.334268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.334343 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.334346 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.334413 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.334456 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260928 09:03:11.334918 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a44230c8b444dbc4e6f13b6d71439" candidate_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1860f0c0ed24939945786d64f0afeff" is_pre_election: true
I20260928 09:03:11.335114 19726 raft_consensus.cc:2454] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d2cec0475a49bb83e7b27ea47f0e8f in term 0.
I20260928 09:03:11.335139 19994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a44230c8b444dbc4e6f13b6d71439" candidate_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab3b9cd024d47898e73308ba197e575" is_pre_election: true
I20260928 09:03:11.335305 19994 raft_consensus.cc:2454] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d2cec0475a49bb83e7b27ea47f0e8f in term 0.
I20260928 09:03:11.335381 19796 leader_election.cc:302] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d2cec0475a49bb83e7b27ea47f0e8f, e1860f0c0ed24939945786d64f0afeff; no voters:
I20260928 09:03:11.335536 20064 raft_consensus.cc:2781] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:03:11.335628 20064 raft_consensus.cc:482] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:03:11.335665 20064 raft_consensus.cc:3037] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:11.336421 20064 raft_consensus.cc:504] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.336588 20064 leader_election.cc:288] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [CANDIDATE]: Term 1 election: Requested vote from peers e1860f0c0ed24939945786d64f0afeff (127.0.98.1:40235), cab3b9cd024d47898e73308ba197e575 (127.0.98.3:45789)
I20260928 09:03:11.336808 19726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a44230c8b444dbc4e6f13b6d71439" candidate_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1860f0c0ed24939945786d64f0afeff"
I20260928 09:03:11.336884 19726 raft_consensus.cc:3037] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:11.337312 19994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a44230c8b444dbc4e6f13b6d71439" candidate_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab3b9cd024d47898e73308ba197e575"
I20260928 09:03:11.337411 19994 raft_consensus.cc:3037] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:11.337886 19726 raft_consensus.cc:2454] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d2cec0475a49bb83e7b27ea47f0e8f in term 1.
I20260928 09:03:11.338124 19796 leader_election.cc:302] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d2cec0475a49bb83e7b27ea47f0e8f, e1860f0c0ed24939945786d64f0afeff; no voters:
I20260928 09:03:11.338303 20064 raft_consensus.cc:2781] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:11.338413 19994 raft_consensus.cc:2454] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d2cec0475a49bb83e7b27ea47f0e8f in term 1.
I20260928 09:03:11.338531 20064 raft_consensus.cc:686] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 1 LEADER]: Becoming Leader. State: Replica: b9d2cec0475a49bb83e7b27ea47f0e8f, State: Running, Role: LEADER
I20260928 09:03:11.338641 20064 consensus_queue.cc:237] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } }
I20260928 09:03:11.339586 19580 catalog_manager.cc:5495] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d2cec0475a49bb83e7b27ea47f0e8f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d2cec0475a49bb83e7b27ea47f0e8f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 } health_report { overall_health: UNKNOWN } } }
I20260928 09:03:11.368625 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.365974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.98.2:36009)
0928 09:03:11.366077 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.366079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.366087 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:11.366124 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.366124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.366144 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.366244 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:11.366402 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.366861 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.367908 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.368108 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.368160 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.368325 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:11.368427 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:11.368490 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.368524 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.368543 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 09:03:11.370513 19994 raft_consensus.cc:1261] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 1 FOLLOWER]: Refusing update from remote peer b9d2cec0475a49bb83e7b27ea47f0e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:11.370541 19726 raft_consensus.cc:1261] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 1 FOLLOWER]: Refusing update from remote peer b9d2cec0475a49bb83e7b27ea47f0e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:11.370926 20067 consensus_queue.cc:1035] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab3b9cd024d47898e73308ba197e575" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:11.371073 20064 consensus_queue.cc:1035] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1860f0c0ed24939945786d64f0afeff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:11.373744 20072 mvcc.cc:205] Tried to move back new op lower bound from 7334241039851700224 to 7334241039724462080. Current Snapshot: MvccSnapshot[applied={T|T < 7334241039851700224}]
I20260928 09:03:11.376623 19580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:11.376761 19580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:11.377550 19580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a4f44464bc4cf28cc91424bcdc34a1: Sending DeleteTablet for 3 replicas of tablet 3e6a44230c8b444dbc4e6f13b6d71439
I20260928 09:03:11.377902 19974 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a44230c8b444dbc4e6f13b6d71439 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34688
I20260928 09:03:11.377954 19840 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a44230c8b444dbc4e6f13b6d71439 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43228
I20260928 09:03:11.377974 19706 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a44230c8b444dbc4e6f13b6d71439 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51808
I20260928 09:03:11.378178 20081 tablet_replica.cc:330] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: stopping tablet replica
I20260928 09:03:11.378275 20081 raft_consensus.cc:2229] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:03:11.378381 20082 tablet_replica.cc:330] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff: stopping tablet replica
I20260928 09:03:11.378400 20081 raft_consensus.cc:2258] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:11.378489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19643
I20260928 09:03:11.378561 20082 raft_consensus.cc:2229] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:11.378669 20082 raft_consensus.cc:2258] T 3e6a44230c8b444dbc4e6f13b6d71439 P e1860f0c0ed24939945786d64f0afeff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:11.379104 20080 tablet_replica.cc:330] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: stopping tablet replica
I20260928 09:03:11.379200 20080 raft_consensus.cc:2229] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:11.379295 20080 raft_consensus.cc:2258] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:11.379704 20080 ts_tablet_manager.cc:1905] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:11.381433 20080 ts_tablet_manager.cc:1918] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:11.381508 20080 log.cc:1196] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/3e6a44230c8b444dbc4e6f13b6d71439
I20260928 09:03:11.381786 20080 ts_tablet_manager.cc:1939] T 3e6a44230c8b444dbc4e6f13b6d71439 P cab3b9cd024d47898e73308ba197e575: Deleting consensus metadata
I20260928 09:03:11.382252 19567 catalog_manager.cc:4841] TS cab3b9cd024d47898e73308ba197e575 (127.0.98.3:45789): tablet 3e6a44230c8b444dbc4e6f13b6d71439 (table test-table [id=87ec49dfdfcb4c5fa94d96548a42faae]) successfully deleted
W20260928 09:03:11.384425 19568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40235 (error 108)
I20260928 09:03:11.384855 20081 ts_tablet_manager.cc:1905] T 3e6a44230c8b444dbc4e6f13b6d71439 P b9d2cec0475a49bb83e7b27ea47f0e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:11.384960 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.384822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40235 (local address 127.0.0.1:51810)
0928 09:03:11.384883 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.384927 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 09:03:11.385093 19568 catalog_manager.cc:4584] TS e1860f0c0ed24939945786d64f0afeff (127.0.98.1:40235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e6a44230c8b444dbc4e6f13b6d71439: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40235: connect: Connection refused (error 111)
I20260928 09:03:11.386204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19777
I20260928 09:03:11.390843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19911
I20260928 09:03:11.390977 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.390826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36009 (local address 127.0.0.1:43248)
0928 09:03:11.390868 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.390896 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 09:03:11.395792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 19549
2026-09-28T09:03:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T09:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:11Z Disabled control of system clock
I20260928 09:03:11.423245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37739
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:11.503319 20087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:11.503547 20087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:11.503568 20087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:11.504688 20087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:11.504722 20087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:11.504735 20087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:11.504748 20087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:11.504760 20087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:11.506033 20087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:11.506225 20087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:11.506470 20087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:03:11.511456 20087 server_base.cc:1034] running on GCE node
W20260928 09:03:11.511415 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.511416 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.511425 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:11.511905 20087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:11.512202 20087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:11.513370 20087 hybrid_clock.cc:648] HybridClock initialized: now 1790586191513337 us; error 66 us; skew 500 ppm
I20260928 09:03:11.514659 20087 webserver.cc:460] Webserver started at http://127.0.98.62:35381/ using document root <none> and password file <none>
I20260928 09:03:11.514969 20087 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:11.515045 20087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:11.515192 20087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:11.516229 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "65d324f6b6fc49f480b355de6fdab1f4"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "5d75d7a9625b181d074a529860704d29"
server_key_iv: "58cf7c914b4fbda5f15f263e915eedfe"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.516582 20087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "65d324f6b6fc49f480b355de6fdab1f4"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "5d75d7a9625b181d074a529860704d29"
server_key_iv: "58cf7c914b4fbda5f15f263e915eedfe"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.517903 20087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:11.518607 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.518793 20087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:11.518934 20087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "65d324f6b6fc49f480b355de6fdab1f4"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "5d75d7a9625b181d074a529860704d29"
server_key_iv: "58cf7c914b4fbda5f15f263e915eedfe"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.519011 20087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:11.530220 20087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:11.530537 20087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:11.530685 20087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:11.534677 20087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37739
I20260928 09:03:11.534710 20153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37739 every 8 connection(s)
I20260928 09:03:11.535050 20087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:11.536043 20154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.537451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20087
I20260928 09:03:11.537586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:11.537791 392 external_mini_cluster.cc:1428] Setting key 775ffd83487132372d6078b24a5a6703
I20260928 09:03:11.539191 20154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4: Bootstrap starting.
I20260928 09:03:11.539824 20154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.540139 20154 log.cc:824] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.541007 20154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4: No bootstrap required, opened a new log
I20260928 09:03:11.543313 20154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d324f6b6fc49f480b355de6fdab1f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37739 } }
I20260928 09:03:11.543460 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.538360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.0.98.62:37739)
0928 09:03:11.538543 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.538558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.538582 (+ 24us) server_negotiation.cc:407] Connection header received
0928 09:03:11.538989 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.538995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.539054 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.539173 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.539444 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.540464 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.541557 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.541974 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.542070 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.542788 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.542792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.542792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:11.542951 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.542982 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:11.542993 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.543043 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.543135 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.543219 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260928 09:03:11.543473 20154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.544051 20154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d324f6b6fc49f480b355de6fdab1f4, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.544179 20154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d324f6b6fc49f480b355de6fdab1f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37739 } }
I20260928 09:03:11.544247 20154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:11.544273 20154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:11.544305 20154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:11.544847 20154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d324f6b6fc49f480b355de6fdab1f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37739 } }
I20260928 09:03:11.544937 20154 leader_election.cc:302] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d324f6b6fc49f480b355de6fdab1f4; no voters:
I20260928 09:03:11.545063 20154 leader_election.cc:288] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:11.545111 20159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:11.545274 20159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [term 1 LEADER]: Becoming Leader. State: Replica: 65d324f6b6fc49f480b355de6fdab1f4, State: Running, Role: LEADER
I20260928 09:03:11.545326 20154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:11.545360 20159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d324f6b6fc49f480b355de6fdab1f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37739 } }
I20260928 09:03:11.545692 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d324f6b6fc49f480b355de6fdab1f4. Latest consensus state: current_term: 1 leader_uuid: "65d324f6b6fc49f480b355de6fdab1f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d324f6b6fc49f480b355de6fdab1f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37739 } } }
I20260928 09:03:11.545701 20161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d324f6b6fc49f480b355de6fdab1f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d324f6b6fc49f480b355de6fdab1f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37739 } } }
I20260928 09:03:11.545773 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:11.545778 20161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:11.546422 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:11.546866 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:11.548349 20165 catalog_manager.cc:1347] Generated new cluster ID: 98ec6d2823744d8f924441a3233670e9
I20260928 09:03:11.548401 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:11.555446 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:11.556140 20165 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:11.558542 20165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4: Generated new TSK 0
I20260928 09:03:11.558706 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:11.560138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37739
--builtin_ntp_servers=127.0.98.20:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:11.645073 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:11.645350 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:11.645402 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:11.647019 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:11.647063 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:11.647089 20178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 09:03:11.648460 20178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:11.648731 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:11.649060 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:11.654050 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.654064 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.654063 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:11.657235 20178 server_base.cc:1034] running on GCE node
I20260928 09:03:11.657471 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:11.657742 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:11.658957 20178 hybrid_clock.cc:648] HybridClock initialized: now 1790586191658934 us; error 38 us; skew 500 ppm
I20260928 09:03:11.660368 20178 webserver.cc:460] Webserver started at http://127.0.98.1:35835/ using document root <none> and password file <none>
I20260928 09:03:11.660694 20178 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:11.660758 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:11.660887 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:11.661983 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data/instance:
uuid: "ecf4d1e7c5eb46978687ad6bcf02f578"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "831514f4890d968de289bb91b927616f"
server_key_iv: "26f3c23068dfd34589af846ebb8f75ba"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.662344 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ecf4d1e7c5eb46978687ad6bcf02f578"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "831514f4890d968de289bb91b927616f"
server_key_iv: "26f3c23068dfd34589af846ebb8f75ba"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.663666 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:11.664438 20192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.664588 20178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:03:11.664695 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal
uuid: "ecf4d1e7c5eb46978687ad6bcf02f578"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "831514f4890d968de289bb91b927616f"
server_key_iv: "26f3c23068dfd34589af846ebb8f75ba"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.664788 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:11.683776 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:11.684115 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:11.684257 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:11.684509 20178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:11.684892 20178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:11.684927 20178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.684950 20178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:11.684998 20178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.690970 20178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38165
I20260928 09:03:11.691020 20305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38165 every 8 connection(s)
I20260928 09:03:11.691339 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/data/info.pb
I20260928 09:03:11.696731 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.692036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41521 (local address 127.0.98.62:37739)
0928 09:03:11.692094 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.692096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.692335 (+ 239us) server_negotiation.cc:407] Connection header received
0928 09:03:11.692990 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.692991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.693008 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.693077 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.693762 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.694277 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.695266 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.695478 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.695584 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.696401 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.696403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.696403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:11.696408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.696424 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:11.696427 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.696506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.696601 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.696648 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:03:11.697033 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.692018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37739 (local address 127.0.98.1:41521)
0928 09:03:11.692216 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.692249 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.692851 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.693159 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.693163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.693171 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:11.693666 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.693671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.694373 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.694374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.695166 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.695168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.695593 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.695594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.695728 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.696254 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:11.696255 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:11.696283 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:11.696307 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:11.696505 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:11.696512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:11.696515 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.696603 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.696725 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 09:03:11.697574 20306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37739
I20260928 09:03:11.697666 20306 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:11.697829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20178
I20260928 09:03:11.697870 20306 heartbeater.cc:511] Master 127.0.98.62:37739 requested a full tablet report, sending...
I20260928 09:03:11.697963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-0/wal/instance
I20260928 09:03:11.698186 392 external_mini_cluster.cc:1428] Setting key a93f3edea327bca7c8a391bb930d4b45
I20260928 09:03:11.698467 20118 ts_manager.cc:194] Registered new tserver with Master: ecf4d1e7c5eb46978687ad6bcf02f578 (127.0.98.1:38165)
I20260928 09:03:11.699306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37739
--builtin_ntp_servers=127.0.98.20:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 09:03:11.699347 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41521
W20260928 09:03:11.781050 20309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:11.781347 20309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:11.781383 20309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:11.782543 20309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:11.782583 20309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:11.782608 20309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 09:03:11.783963 20309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:11.784147 20309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:11.784392 20309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:11.789391 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.789412 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.789403 20314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:11.789515 20309 server_base.cc:1034] running on GCE node
I20260928 09:03:11.789836 20309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:11.790102 20309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:11.791262 20309 hybrid_clock.cc:648] HybridClock initialized: now 1790586191791241 us; error 34 us; skew 500 ppm
I20260928 09:03:11.792513 20309 webserver.cc:460] Webserver started at http://127.0.98.2:44543/ using document root <none> and password file <none>
I20260928 09:03:11.792809 20309 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:11.792879 20309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:11.793023 20309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:11.794202 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data/instance:
uuid: "545bf8da5ec74cdea74bd89baa2f654d"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "fa8625676946ceb21c79f5cecbd387b7"
server_key_iv: "a0435c746edfbb8b73e9ddcd244151fd"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.794575 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal/instance:
uuid: "545bf8da5ec74cdea74bd89baa2f654d"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "fa8625676946ceb21c79f5cecbd387b7"
server_key_iv: "a0435c746edfbb8b73e9ddcd244151fd"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.795838 20309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:11.796573 20323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.796758 20309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:11.797035 20309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal
uuid: "545bf8da5ec74cdea74bd89baa2f654d"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "fa8625676946ceb21c79f5cecbd387b7"
server_key_iv: "a0435c746edfbb8b73e9ddcd244151fd"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.797140 20309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:11.817381 20309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:11.817725 20309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:11.817884 20309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:11.818128 20309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:11.818463 20309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:11.818517 20309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.818565 20309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:11.818589 20309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.825201 20309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39379
I20260928 09:03:11.825241 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39379 every 8 connection(s)
I20260928 09:03:11.825598 20309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/data/info.pb
I20260928 09:03:11.830704 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.826329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48677 (local address 127.0.98.62:37739)
0928 09:03:11.826387 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.826389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.826644 (+ 255us) server_negotiation.cc:407] Connection header received
0928 09:03:11.827122 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.827123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.827133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.827186 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.827828 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.828314 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.829400 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.829579 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.829628 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.830466 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.830468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.830468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:11.830472 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.830485 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:11.830488 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.830530 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.830623 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.830646 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 09:03:11.831003 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.826345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37739 (local address 127.0.98.2:48677)
0928 09:03:11.826517 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.826551 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.827032 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.827247 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.827250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.827259 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:11.827752 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.827756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.828422 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.828423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.829323 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.829325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.829647 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.829648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.829760 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.830313 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:11.830314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:11.830344 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:11.830369 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:11.830546 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:11.830550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:11.830553 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.830611 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.830719 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260928 09:03:11.831542 20437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37739
I20260928 09:03:11.831634 20437 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:11.831779 20437 heartbeater.cc:511] Master 127.0.98.62:37739 requested a full tablet report, sending...
I20260928 09:03:11.832157 20118 ts_manager.cc:194] Registered new tserver with Master: 545bf8da5ec74cdea74bd89baa2f654d (127.0.98.2:39379)
I20260928 09:03:11.832485 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48677
I20260928 09:03:11.834592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20309
I20260928 09:03:11.834683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal/instance
I20260928 09:03:11.834895 392 external_mini_cluster.cc:1428] Setting key d0ac0f4d436ce4983653dfe4e1f9ad9d
I20260928 09:03:11.835829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37739
--builtin_ntp_servers=127.0.98.20:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 09:03:11.917171 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:11.917420 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:11.917455 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:11.918591 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:11.918658 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:11.918700 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 09:03:11.920048 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:11.920331 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:11.920647 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:11.926028 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.926014 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:11.926012 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:11.926275 20440 server_base.cc:1034] running on GCE node
I20260928 09:03:11.926496 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:11.926748 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:11.927919 20440 hybrid_clock.cc:648] HybridClock initialized: now 1790586191927907 us; error 29 us; skew 500 ppm
I20260928 09:03:11.929332 20440 webserver.cc:460] Webserver started at http://127.0.98.3:43679/ using document root <none> and password file <none>
I20260928 09:03:11.929586 20440 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:11.929657 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:11.929811 20440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:11.930961 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f39b19820af42fa8ce7a428a951b6ec"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "c4406afabe2f7fbb6f6ddffeeafba783"
server_key_iv: "4313894e838c710238a4123a0f55fce9"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.931334 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f39b19820af42fa8ce7a428a951b6ec"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "c4406afabe2f7fbb6f6ddffeeafba783"
server_key_iv: "4313894e838c710238a4123a0f55fce9"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.932678 20440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:11.933471 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.933642 20440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:11.933779 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal
uuid: "0f39b19820af42fa8ce7a428a951b6ec"
format_stamp: "Formatted at 2026-09-28 09:03:11 on dist-test-slave-286n"
server_key: "c4406afabe2f7fbb6f6ddffeeafba783"
server_key_iv: "4313894e838c710238a4123a0f55fce9"
server_key_version: "encryptionkey@0"
I20260928 09:03:11.933853 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:11.951020 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:11.951383 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:11.951551 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:11.951804 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 09:03:11.952147 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 09:03:11.952199 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.952239 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 09:03:11.952281 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:11.958066 20440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46173
I20260928 09:03:11.958108 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46173 every 8 connection(s)
I20260928 09:03:11.958456 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/data/info.pb
I20260928 09:03:11.960593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20440
I20260928 09:03:11.960760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal/instance
I20260928 09:03:11.960989 392 external_mini_cluster.cc:1428] Setting key ee6a40d0940555914547f5d4c0d18da9
I20260928 09:03:11.964262 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.959315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57535 (local address 127.0.98.62:37739)
0928 09:03:11.959382 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.959384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.959651 (+ 267us) server_negotiation.cc:407] Connection header received
0928 09:03:11.960116 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.960118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.960129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.960200 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.961069 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.961888 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.962944 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.963133 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.963199 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.964018 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.964019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.964020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:11.964024 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.964038 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:11.964042 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.964086 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.964173 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.964196 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:03:11.964512 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.959291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37739 (local address 127.0.98.3:57535)
0928 09:03:11.959474 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.959517 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.960001 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.960279 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.960283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.960291 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:11.960964 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.960970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.962011 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.962012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.962845 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.962847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.963232 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.963234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.963360 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.963885 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 09:03:11.963885 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:11.963914 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 09:03:11.963937 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:11.964099 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:11.964102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:11.964105 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.964158 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.964245 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 09:03:11.965046 20568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37739
I20260928 09:03:11.965173 20568 heartbeater.cc:464] Registering TS with master...
I20260928 09:03:11.965397 20568 heartbeater.cc:511] Master 127.0.98.62:37739 requested a full tablet report, sending...
I20260928 09:03:11.965890 20118 ts_manager.cc:194] Registered new tserver with Master: 0f39b19820af42fa8ce7a428a951b6ec (127.0.98.3:46173)
I20260928 09:03:11.966266 20118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57535
I20260928 09:03:11.973158 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 09:03:11.977908 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.974693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.98.62:37739)
0928 09:03:11.974755 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.974757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.974918 (+ 161us) server_negotiation.cc:407] Connection header received
0928 09:03:11.974990 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.974991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.975001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.975059 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:11.975256 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.975814 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.976602 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.976841 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.976929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.977657 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:11.977659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:11.977659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:11.977664 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:11.977677 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:11.977682 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:11.977734 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.977792 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.977816 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:03:11.979544 20118 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 09:03:11.981202 20118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 09:03:11.987805 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.983354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38165 (local address 127.0.0.1:60898)
0928 09:03:11.983734 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.983778 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.983856 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.984106 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.984116 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.984120 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.984198 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.984200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.985594 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.985595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.986811 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.986813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.987509 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.987510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.987584 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.987597 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.987673 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.987732 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":53,"thread_start_us":58,"threads_started":1}
I20260928 09:03:11.987816 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.983598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.0.98.1:38165)
0928 09:03:11.983791 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.983797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.983831 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:03:11.983967 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.983970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.983987 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.984074 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.984307 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.985507 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.986898 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.987418 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.987508 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.987600 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.987686 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.987725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260928 09:03:11.988242 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.983364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46173 (local address 127.0.0.1:57368)
0928 09:03:11.984159 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.984168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.984219 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.984473 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.984474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.984476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.984527 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.984528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.985439 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.985441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.986822 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.986825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.987813 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.987814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.987867 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.987871 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.988118 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.988157 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260928 09:03:11.988467 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.983532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.98.3:46173)
0928 09:03:11.983740 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.983746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.984278 (+ 532us) server_negotiation.cc:407] Connection header received
0928 09:03:11.984323 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.984326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.984342 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.984410 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.984602 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.985351 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.986923 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.987636 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.987697 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.987781 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.988346 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.988386 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260928 09:03:11.988584 20240 tablet_service.cc:1467] Processing CreateTablet for tablet 1676227aa7a146b499b8aa155cf6398e (DEFAULT_TABLE table=test-table [id=39c19a27523a4283a117d5d33c5d6c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:11.988926 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.983354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39379 (local address 127.0.0.1:51324)
0928 09:03:11.983992 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:11.984002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:11.984064 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:11.984315 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:11.984316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:11.984319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:11.984355 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.984356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.985678 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.985679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.987571 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:11.987574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.988704 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.988705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:11.988769 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.988772 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:11.988823 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 09:03:11.988861 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":851,"thread_start_us":103,"threads_started":1}
I20260928 09:03:11.989008 20240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1676227aa7a146b499b8aa155cf6398e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.989135 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:11.983534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.98.2:39379)
0928 09:03:11.983774 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:11.983778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:11.984084 (+ 306us) server_negotiation.cc:407] Connection header received
0928 09:03:11.984128 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:11.984130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:11.984146 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:11.984243 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:11.984430 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.985557 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.987673 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:11.988545 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:11.988618 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:11.988742 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:11.989014 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 09:03:11.989056 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
I20260928 09:03:11.989180 20502 tablet_service.cc:1467] Processing CreateTablet for tablet 1676227aa7a146b499b8aa155cf6398e (DEFAULT_TABLE table=test-table [id=39c19a27523a4283a117d5d33c5d6c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:11.989537 20502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1676227aa7a146b499b8aa155cf6398e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.989708 20371 tablet_service.cc:1467] Processing CreateTablet for tablet 1676227aa7a146b499b8aa155cf6398e (DEFAULT_TABLE table=test-table [id=39c19a27523a4283a117d5d33c5d6c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 09:03:11.990012 20371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1676227aa7a146b499b8aa155cf6398e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:11.992861 20583 tablet_bootstrap.cc:492] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Bootstrap starting.
I20260928 09:03:11.992892 20582 tablet_bootstrap.cc:492] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578: Bootstrap starting.
I20260928 09:03:11.993108 20584 tablet_bootstrap.cc:492] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Bootstrap starting.
I20260928 09:03:11.993567 20582 tablet_bootstrap.cc:654] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.993618 20583 tablet_bootstrap.cc:654] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.993706 20584 tablet_bootstrap.cc:654] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:11.993925 20582 log.cc:824] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.993923 20583 log.cc:824] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.994014 20584 log.cc:824] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:11.994697 20583 tablet_bootstrap.cc:492] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: No bootstrap required, opened a new log
I20260928 09:03:11.994776 20584 tablet_bootstrap.cc:492] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: No bootstrap required, opened a new log
I20260928 09:03:11.994779 20583 ts_tablet_manager.cc:1397] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:03:11.994851 20584 ts_tablet_manager.cc:1397] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:03:11.994892 20582 tablet_bootstrap.cc:492] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578: No bootstrap required, opened a new log
I20260928 09:03:11.994958 20582 ts_tablet_manager.cc:1397] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 09:03:11.996851 20583 raft_consensus.cc:348] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:11.997002 20583 raft_consensus.cc:374] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.997044 20583 raft_consensus.cc:729] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f39b19820af42fa8ce7a428a951b6ec, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.997150 20583 consensus_queue.cc:260] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:11.997430 20583 ts_tablet_manager.cc:1428] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 09:03:11.997450 20568 heartbeater.cc:503] Master 127.0.98.62:37739 was elected leader, sending a full tablet report...
I20260928 09:03:11.997466 20584 raft_consensus.cc:348] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:11.997614 20584 raft_consensus.cc:374] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.997646 20584 raft_consensus.cc:729] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545bf8da5ec74cdea74bd89baa2f654d, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.997800 20584 consensus_queue.cc:260] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:11.998054 20584 ts_tablet_manager.cc:1428] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 09:03:11.998093 20437 heartbeater.cc:503] Master 127.0.98.62:37739 was elected leader, sending a full tablet report...
I20260928 09:03:11.998085 20582 raft_consensus.cc:348] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:11.998221 20582 raft_consensus.cc:374] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:11.998245 20582 raft_consensus.cc:729] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf4d1e7c5eb46978687ad6bcf02f578, State: Initialized, Role: FOLLOWER
I20260928 09:03:11.998376 20582 consensus_queue.cc:260] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:11.998653 20582 ts_tablet_manager.cc:1428] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 09:03:11.998766 20306 heartbeater.cc:503] Master 127.0.98.62:37739 was elected leader, sending a full tablet report...
W20260928 09:03:12.076184 20438 tablet.cc:2369] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 09:03:12.088352 20589 raft_consensus.cc:482] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 09:03:12.088474 20589 raft_consensus.cc:504] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:12.088960 20589 leader_election.cc:288] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f39b19820af42fa8ce7a428a951b6ec (127.0.98.3:46173), ecf4d1e7c5eb46978687ad6bcf02f578 (127.0.98.1:38165)
I20260928 09:03:12.091817 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:12.089224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57651 (local address 127.0.98.3:46173)
0928 09:03:12.089264 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:12.089265 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:12.089282 (+ 17us) server_negotiation.cc:407] Connection header received
0928 09:03:12.089372 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:12.089373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:12.089386 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:12.089450 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:12.089669 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.090126 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.091112 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.091469 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.091553 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:12.091563 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:12.091713 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 09:03:12.091750 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:12.091817 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:12.089156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46173 (local address 127.0.98.2:57651)
0928 09:03:12.089230 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:12.089242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:12.089291 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:12.089455 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:12.089456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:12.089457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:12.089534 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:12.089536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.090218 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.090219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:12.091007 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:12.091010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.091553 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.091554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:12.091630 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:12.091633 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:12.091695 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 09:03:12.091740 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":46}
I20260928 09:03:12.091897 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:12.089243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44961 (local address 127.0.98.1:38165)
0928 09:03:12.089291 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:12.089294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:12.089417 (+ 123us) server_negotiation.cc:407] Connection header received
0928 09:03:12.089496 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:12.089497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:12.089509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:12.089582 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 09:03:12.089777 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.090237 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.091167 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.091532 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.091615 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:12.091624 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:12.091782 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 09:03:12.091818 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:03:12.091950 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:12.089176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38165 (local address 127.0.98.2:44961)
0928 09:03:12.089348 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:12.089356 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:12.089431 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:12.089585 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:12.089585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:12.089587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 09:03:12.089659 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:12.089660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.090320 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.090321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:12.091073 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:12.091075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.091618 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.091619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:12.091706 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:12.091709 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 09:03:12.091777 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 09:03:12.091814 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260928 09:03:12.092176 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1676227aa7a146b499b8aa155cf6398e" candidate_uuid: "545bf8da5ec74cdea74bd89baa2f654d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f39b19820af42fa8ce7a428a951b6ec" is_pre_election: true
I20260928 09:03:12.092300 20522 raft_consensus.cc:2454] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545bf8da5ec74cdea74bd89baa2f654d in term 0.
I20260928 09:03:12.092569 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1676227aa7a146b499b8aa155cf6398e" candidate_uuid: "545bf8da5ec74cdea74bd89baa2f654d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" is_pre_election: true
I20260928 09:03:12.092741 20260 raft_consensus.cc:2454] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545bf8da5ec74cdea74bd89baa2f654d in term 0.
I20260928 09:03:12.092886 20324 leader_election.cc:302] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f39b19820af42fa8ce7a428a951b6ec, 545bf8da5ec74cdea74bd89baa2f654d; no voters:
I20260928 09:03:12.093053 20589 raft_consensus.cc:2781] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 09:03:12.093107 20589 raft_consensus.cc:482] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 09:03:12.093124 20589 raft_consensus.cc:3037] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:12.093842 20589 raft_consensus.cc:504] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:12.093993 20589 leader_election.cc:288] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [CANDIDATE]: Term 1 election: Requested vote from peers 0f39b19820af42fa8ce7a428a951b6ec (127.0.98.3:46173), ecf4d1e7c5eb46978687ad6bcf02f578 (127.0.98.1:38165)
I20260928 09:03:12.094127 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1676227aa7a146b499b8aa155cf6398e" candidate_uuid: "545bf8da5ec74cdea74bd89baa2f654d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578"
I20260928 09:03:12.094129 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1676227aa7a146b499b8aa155cf6398e" candidate_uuid: "545bf8da5ec74cdea74bd89baa2f654d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f39b19820af42fa8ce7a428a951b6ec"
I20260928 09:03:12.094210 20522 raft_consensus.cc:3037] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:12.094206 20260 raft_consensus.cc:3037] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:12.094921 20260 raft_consensus.cc:2454] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545bf8da5ec74cdea74bd89baa2f654d in term 1.
I20260928 09:03:12.095016 20522 raft_consensus.cc:2454] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545bf8da5ec74cdea74bd89baa2f654d in term 1.
I20260928 09:03:12.095105 20327 leader_election.cc:302] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545bf8da5ec74cdea74bd89baa2f654d, ecf4d1e7c5eb46978687ad6bcf02f578; no voters:
I20260928 09:03:12.095224 20589 raft_consensus.cc:2781] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:12.095381 20589 raft_consensus.cc:686] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 1 LEADER]: Becoming Leader. State: Replica: 545bf8da5ec74cdea74bd89baa2f654d, State: Running, Role: LEADER
I20260928 09:03:12.095484 20589 consensus_queue.cc:237] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } }
I20260928 09:03:12.096339 20118 catalog_manager.cc:5495] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d reported cstate change: term changed from 0 to 1, leader changed from <none> to 545bf8da5ec74cdea74bd89baa2f654d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "545bf8da5ec74cdea74bd89baa2f654d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "545bf8da5ec74cdea74bd89baa2f654d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 } health_report { overall_health: UNKNOWN } } }
I20260928 09:03:12.103325 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:12.100719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.0.98.2:39379)
0928 09:03:12.100778 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:12.100780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:12.100805 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:03:12.100924 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:12.100925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:12.100945 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:12.101015 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:12.101165 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.101736 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.102510 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:12.102725 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:12.102804 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:12.102971 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:12.103086 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:12.103184 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:12.103203 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 09:03:12.103222 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 09:03:12.105258 20260 raft_consensus.cc:1261] T 1676227aa7a146b499b8aa155cf6398e P ecf4d1e7c5eb46978687ad6bcf02f578 [term 1 FOLLOWER]: Refusing update from remote peer 545bf8da5ec74cdea74bd89baa2f654d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:12.105259 20522 raft_consensus.cc:1261] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 1 FOLLOWER]: Refusing update from remote peer 545bf8da5ec74cdea74bd89baa2f654d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:12.105641 20592 consensus_queue.cc:1035] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f39b19820af42fa8ce7a428a951b6ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:12.105799 20589 consensus_queue.cc:1035] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf4d1e7c5eb46978687ad6bcf02f578" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:12.108094 20595 mvcc.cc:205] Tried to move back new op lower bound from 7334241042860314624 to 7334241042824232960. Current Snapshot: MvccSnapshot[applied={T|T < 7334241042860314624}]
I20260928 09:03:12.108582 20596 mvcc.cc:205] Tried to move back new op lower bound from 7334241042860314624 to 7334241042824232960. Current Snapshot: MvccSnapshot[applied={T|T < 7334241042860314624}]
I20260928 09:03:12.108762 20597 mvcc.cc:205] Tried to move back new op lower bound from 7334241042860314624 to 7334241042824232960. Current Snapshot: MvccSnapshot[applied={T|T < 7334241042860314624}]
I20260928 09:03:12.110673 20118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:12.110771 20118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 09:03:12.111519 20118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65d324f6b6fc49f480b355de6fdab1f4: Sending DeleteTablet for 3 replicas of tablet 1676227aa7a146b499b8aa155cf6398e
I20260928 09:03:12.111835 20371 tablet_service.cc:1514] Processing DeleteTablet for tablet 1676227aa7a146b499b8aa155cf6398e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:12 UTC) from {username='slave'} at 127.0.0.1:51324
I20260928 09:03:12.111857 20240 tablet_service.cc:1514] Processing DeleteTablet for tablet 1676227aa7a146b499b8aa155cf6398e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:12 UTC) from {username='slave'} at 127.0.0.1:60898
I20260928 09:03:12.112057 20606 tablet_replica.cc:330] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: stopping tablet replica
I20260928 09:03:12.112183 20502 tablet_service.cc:1514] Processing DeleteTablet for tablet 1676227aa7a146b499b8aa155cf6398e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 09:03:12 UTC) from {username='slave'} at 127.0.0.1:57368
I20260928 09:03:12.112371 20607 tablet_replica.cc:330] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: stopping tablet replica
I20260928 09:03:12.112421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20178
I20260928 09:03:12.112464 20607 raft_consensus.cc:2229] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 09:03:12.113041 20606 raft_consensus.cc:2229] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 1 LEADER]: Raft consensus shutting down.
I20260928 09:03:12.113077 20607 raft_consensus.cc:2258] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:12.113243 20606 raft_consensus.cc:2258] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 09:03:12.113567 20607 ts_tablet_manager.cc:1905] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:12.113725 20606 ts_tablet_manager.cc:1905] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 09:03:12.115291 20606 ts_tablet_manager.cc:1918] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:12.115293 20607 ts_tablet_manager.cc:1918] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 09:03:12.115365 20607 log.cc:1196] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-2/wal/wals/1676227aa7a146b499b8aa155cf6398e
I20260928 09:03:12.115514 20606 log.cc:1196] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Deleting WAL directory at /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585920139418-392-0/minicluster-data/ts-1/wal/wals/1676227aa7a146b499b8aa155cf6398e
I20260928 09:03:12.115671 20607 ts_tablet_manager.cc:1939] T 1676227aa7a146b499b8aa155cf6398e P 0f39b19820af42fa8ce7a428a951b6ec: Deleting consensus metadata
I20260928 09:03:12.115912 20606 ts_tablet_manager.cc:1939] T 1676227aa7a146b499b8aa155cf6398e P 545bf8da5ec74cdea74bd89baa2f654d: Deleting consensus metadata
I20260928 09:03:12.116266 20102 catalog_manager.cc:4841] TS 0f39b19820af42fa8ce7a428a951b6ec (127.0.98.3:46173): tablet 1676227aa7a146b499b8aa155cf6398e (table test-table [id=39c19a27523a4283a117d5d33c5d6c65]) successfully deleted
I20260928 09:03:12.116320 20104 catalog_manager.cc:4841] TS 545bf8da5ec74cdea74bd89baa2f654d (127.0.98.2:39379): tablet 1676227aa7a146b499b8aa155cf6398e (table test-table [id=39c19a27523a4283a117d5d33c5d6c65]) successfully deleted
W20260928 09:03:12.120563 20105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38165 (error 108)
I20260928 09:03:12.121134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20309
I20260928 09:03:12.121177 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:12.121004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38165 (local address 127.0.0.1:60908)
0928 09:03:12.121068 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:12.121147 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 09:03:12.122105 20105 catalog_manager.cc:4584] TS ecf4d1e7c5eb46978687ad6bcf02f578 (127.0.98.1:38165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1676227aa7a146b499b8aa155cf6398e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38165: connect: Connection refused (error 111)
I20260928 09:03:12.126595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20440
I20260928 09:03:12.131546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20087
2026-09-28T09:03:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (735 ms)
[----------] 6 tests from EncryptionPolicyTest (8845 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:12 dist-test-slave-286n krb5kdc[20611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:12 dist-test-slave-286n krb5kdc[20611](info): set up 2 sockets
Sep 28 09:03:12 dist-test-slave-286n krb5kdc[20611](info): commencing operation
krb5kdc: starting...
W20260928 09:03:13.180702 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:13 dist-test-slave-286n krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:13.300611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:13.383212 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:13.383497 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:13.383541 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:13.384766 20624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:13.384836 20624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:13.384859 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:13.384881 20624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:13.384912 20624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:13.386407 20624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:13.386691 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:13.386932 20624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:13.391943 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:13.391937 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:13.392004 20632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:13.392025 20624 server_base.cc:1034] running on GCE node
I20260928 09:03:13.392473 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:13.392781 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:13.393925 20624 hybrid_clock.cc:648] HybridClock initialized: now 1790586193393894 us; error 49 us; skew 500 ppm
Sep 28 09:03:13 dist-test-slave-286n krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:13.396332 20624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:13.396791 20624 webserver.cc:460] Webserver started at http://127.0.0.1:34627/ using document root <none> and password file <none>
I20260928 09:03:13.396979 20624 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:13.397029 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:13.397125 20624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:13.398193 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "12972e501dae4c8c9eeb431260a69bce"
format_stamp: "Formatted at 2026-09-28 09:03:13 on dist-test-slave-286n"
server_key: "1ee1cef3ac65d121f914b8bf5986b19b"
server_key_iv: "1dea3a98916bd88da1c963da4b7aee5f"
server_key_version: "encryptionkey@0"
I20260928 09:03:13.398501 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "12972e501dae4c8c9eeb431260a69bce"
format_stamp: "Formatted at 2026-09-28 09:03:13 on dist-test-slave-286n"
server_key: "1ee1cef3ac65d121f914b8bf5986b19b"
server_key_iv: "1dea3a98916bd88da1c963da4b7aee5f"
server_key_version: "encryptionkey@0"
I20260928 09:03:13.399798 20624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:13.400494 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:13.400674 20624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:13.400766 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "12972e501dae4c8c9eeb431260a69bce"
format_stamp: "Formatted at 2026-09-28 09:03:13 on dist-test-slave-286n"
server_key: "1ee1cef3ac65d121f914b8bf5986b19b"
server_key_iv: "1dea3a98916bd88da1c963da4b7aee5f"
server_key_version: "encryptionkey@0"
I20260928 09:03:13.400821 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:13.419764 20624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:13.422236 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:13.422351 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:13.426326 20624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33539
I20260928 09:03:13.426411 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33539 every 8 connection(s)
I20260928 09:03:13.426771 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:13.427690 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:13.430073 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce: Bootstrap starting.
I20260928 09:03:13.430668 20692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:13.430965 20692 log.cc:824] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:13.431689 20692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce: No bootstrap required, opened a new log
I20260928 09:03:13.433423 20692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12972e501dae4c8c9eeb431260a69bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260928 09:03:13.433588 20692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:13.433633 20692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12972e501dae4c8c9eeb431260a69bce, State: Initialized, Role: FOLLOWER
I20260928 09:03:13.433807 20692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12972e501dae4c8c9eeb431260a69bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260928 09:03:13.433893 20692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:13.433930 20692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:13.433982 20692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:13.434535 20692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12972e501dae4c8c9eeb431260a69bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260928 09:03:13.434660 20692 leader_election.cc:302] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12972e501dae4c8c9eeb431260a69bce; no voters:
I20260928 09:03:13.434855 20692 leader_election.cc:288] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:13.434916 20695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:13.435124 20692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:13.435461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20624
I20260928 09:03:13.435631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:13.435850 392 external_mini_cluster.cc:1428] Setting key 34cbe4d9864ffb0bd33e929573ac9bb1
I20260928 09:03:13.435143 20695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [term 1 LEADER]: Becoming Leader. State: Replica: 12972e501dae4c8c9eeb431260a69bce, State: Running, Role: LEADER
I20260928 09:03:13.436503 20695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12972e501dae4c8c9eeb431260a69bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260928 09:03:13.437192 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12972e501dae4c8c9eeb431260a69bce. Latest consensus state: current_term: 1 leader_uuid: "12972e501dae4c8c9eeb431260a69bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12972e501dae4c8c9eeb431260a69bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } } }
I20260928 09:03:13.437848 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:13.437373 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12972e501dae4c8c9eeb431260a69bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12972e501dae4c8c9eeb431260a69bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } } }
I20260928 09:03:13.438038 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce [sys.catalog]: This master's current role is: LEADER
W20260928 09:03:13.437628 20711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:13.438553 20711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:13.438663 20714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:13.438855 20714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:13.441100 20714 catalog_manager.cc:1347] Generated new cluster ID: 5faabac0af4b42139e4c328546046a5b
I20260928 09:03:13.441159 20714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:13 dist-test-slave-286n krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:03:13.448740 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:13.436390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59087 (local address 127.0.0.1:33539)
0928 09:03:13.436556 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:13.436568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:13.436873 (+ 305us) server_negotiation.cc:407] Connection header received
0928 09:03:13.437444 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:13.437453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:13.437550 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:13.437734 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:13.438542 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:13.439957 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:13.440819 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:13.441472 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:13.441589 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:13.444705 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:13.444710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:13.444711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:13.444727 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:13.446859 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:13.447358 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:13.447361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:13.447362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:13.447490 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:13.447740 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:13.447741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:13.447741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:13.448065 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:13.448238 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:13.448349 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 09:03:13.448469 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 09:03:13.449781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:13.459324 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:13.451368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42301 (local address 127.0.0.1:33539)
0928 09:03:13.451423 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:13.451425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:13.451650 (+ 225us) server_negotiation.cc:407] Connection header received
0928 09:03:13.451696 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:13.451697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:13.451707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:13.451767 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:13.452423 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:13.453273 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:13.454283 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:13.454607 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:13.454680 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:13.456184 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:13.456185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:13.456185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:13.456193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:13.458031 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:13.458473 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:13.458474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:13.458475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:13.458537 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:13.458774 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:13.458775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:13.458775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:13.458916 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:13.459034 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:13.459161 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 09:03:13.459225 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 09:03:13.462481 20714 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:13.463047 20714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:13.466738 20714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12972e501dae4c8c9eeb431260a69bce: Generated new TSK 0
I20260928 09:03:13.466950 20714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:13.471064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20624
W20260928 09:03:13.475040 20609 connection.cc:535] client connection to 127.0.0.1:33539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33539): Connection reset by peer (error 104)
2026-09-28T09:03:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:13 dist-test-slave-286n krb5kdc[20723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:13 dist-test-slave-286n krb5kdc[20723](info): set up 2 sockets
Sep 28 09:03:13 dist-test-slave-286n krb5kdc[20723](info): commencing operation
krb5kdc: starting...
W20260928 09:03:14.536814 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:14 dist-test-slave-286n krb5kdc[20723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:14.659745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:14.742468 20736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:14.742671 20736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:14.742697 20736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:14.743772 20736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:14.743810 20736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:14.743826 20736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:14.743841 20736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:14.743855 20736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:14.746049 20736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:14.746569 20736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:14.746868 20736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:14.752045 20742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:14.752066 20744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:14.752218 20736 server_base.cc:1034] running on GCE node
W20260928 09:03:14.752067 20741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:14.752604 20736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:14.752972 20736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:14.754144 20736 hybrid_clock.cc:648] HybridClock initialized: now 1790586194754124 us; error 39 us; skew 500 ppm
Sep 28 09:03:14 dist-test-slave-286n krb5kdc[20723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:14.756778 20736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:14.757264 20736 webserver.cc:460] Webserver started at http://127.0.0.1:37559/ using document root <none> and password file <none>
I20260928 09:03:14.757520 20736 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:14.757588 20736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:14.757740 20736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:14.758793 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "f3f333f92cb74de69de8c9f285ae774c"
format_stamp: "Formatted at 2026-09-28 09:03:14 on dist-test-slave-286n"
server_key: "959733e45ead206d0f48ca227802798f"
server_key_iv: "5f779a782792f35947228351e4fb76df"
server_key_version: "encryptionkey@0"
I20260928 09:03:14.759150 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3f333f92cb74de69de8c9f285ae774c"
format_stamp: "Formatted at 2026-09-28 09:03:14 on dist-test-slave-286n"
server_key: "959733e45ead206d0f48ca227802798f"
server_key_iv: "5f779a782792f35947228351e4fb76df"
server_key_version: "encryptionkey@0"
I20260928 09:03:14.760459 20736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:14.761242 20751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:14.761413 20736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:14.761510 20736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "f3f333f92cb74de69de8c9f285ae774c"
format_stamp: "Formatted at 2026-09-28 09:03:14 on dist-test-slave-286n"
server_key: "959733e45ead206d0f48ca227802798f"
server_key_iv: "5f779a782792f35947228351e4fb76df"
server_key_version: "encryptionkey@0"
I20260928 09:03:14.761595 20736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:14.780488 20736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:14.782509 20736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:14.782698 20736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:14.786857 20736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36167
I20260928 09:03:14.786895 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36167 every 8 connection(s)
I20260928 09:03:14.787261 20736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:14.788113 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:14.790547 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c: Bootstrap starting.
I20260928 09:03:14.791124 20804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:14.791415 20804 log.cc:824] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:14.792094 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c: No bootstrap required, opened a new log
I20260928 09:03:14.793789 20804 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f333f92cb74de69de8c9f285ae774c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36167 } }
I20260928 09:03:14.793932 20804 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:14.793955 20804 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f333f92cb74de69de8c9f285ae774c, State: Initialized, Role: FOLLOWER
I20260928 09:03:14.794126 20804 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f333f92cb74de69de8c9f285ae774c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36167 } }
I20260928 09:03:14.794185 20804 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:14.794206 20804 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:14.794286 20804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:14.794476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20736
I20260928 09:03:14.794603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:14.794780 392 external_mini_cluster.cc:1428] Setting key bfbd19ce74870a472562e008522853a5
I20260928 09:03:14.795035 20804 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f333f92cb74de69de8c9f285ae774c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36167 } }
I20260928 09:03:14.795153 20804 leader_election.cc:302] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f333f92cb74de69de8c9f285ae774c; no voters:
I20260928 09:03:14.795318 20804 leader_election.cc:288] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:14.795363 20808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:14.795468 20808 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [term 1 LEADER]: Becoming Leader. State: Replica: f3f333f92cb74de69de8c9f285ae774c, State: Running, Role: LEADER
I20260928 09:03:14.795589 20804 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:14.795615 20808 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f333f92cb74de69de8c9f285ae774c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36167 } }
I20260928 09:03:14.796015 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f333f92cb74de69de8c9f285ae774c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f333f92cb74de69de8c9f285ae774c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36167 } } }
I20260928 09:03:14.796099 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:14.796207 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f333f92cb74de69de8c9f285ae774c. Latest consensus state: current_term: 1 leader_uuid: "f3f333f92cb74de69de8c9f285ae774c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f333f92cb74de69de8c9f285ae774c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36167 } } }
I20260928 09:03:14.796325 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:14.796823 20813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:14.797366 20813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:14.799054 20813 catalog_manager.cc:1347] Generated new cluster ID: 4f2efe7926c54811b0795b9dfada0e00
I20260928 09:03:14.799104 20813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:14.801115 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:14.795277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44589 (local address 127.0.0.1:36167)
0928 09:03:14.795538 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:14.795548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:14.795579 (+ 31us) server_negotiation.cc:407] Connection header received
0928 09:03:14.795745 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:14.795752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:14.795814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:14.795961 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:14.796237 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:14.797671 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:14.798749 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:14.799353 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:14.799468 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:14.800257 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:14.800261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:14.800262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:14.800519 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:14.800562 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:14.800577 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:14.800693 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:14.800781 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 09:03:14.800858 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260928 09:03:14.809067 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:14.811378 20813 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:14.811869 20813 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:14.820502 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:14.810561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45881 (local address 127.0.0.1:36167)
0928 09:03:14.810599 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:14.810601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:14.816891 (+ 6290us) server_negotiation.cc:407] Connection header received
0928 09:03:14.816953 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:14.816954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:14.816965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:14.817015 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:14.817187 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:14.817948 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:14.819033 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:14.819345 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:14.819406 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:14.820221 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:14.820223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:14.820223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:14.820228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:14.820244 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:14.820248 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:14.820312 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:14.820399 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 09:03:14.820413 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:03:14.822204 20813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f333f92cb74de69de8c9f285ae774c: Generated new TSK 0
I20260928 09:03:14.822394 20813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:14.828253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20736
W20260928 09:03:14.832211 20721 connection.cc:535] client connection to 127.0.0.1:36167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36167): Connection reset by peer (error 104)
2026-09-28T09:03:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:14 dist-test-slave-286n krb5kdc[20835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:14 dist-test-slave-286n krb5kdc[20835](info): set up 2 sockets
Sep 28 09:03:14 dist-test-slave-286n krb5kdc[20835](info): commencing operation
krb5kdc: starting...
W20260928 09:03:15.864740 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:15 dist-test-slave-286n krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:15.983702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:16.065704 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:16.065972 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:16.066016 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:16.067198 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:16.067273 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:16.067297 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:16.067318 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:16.067348 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:16.068903 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:16.069193 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:16.069437 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:16.074681 20856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:16.074716 20854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:16.074851 20848 server_base.cc:1034] running on GCE node
W20260928 09:03:16.074733 20853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:16.075193 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:16.075521 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:16.076701 20848 hybrid_clock.cc:648] HybridClock initialized: now 1790586196076690 us; error 48 us; skew 500 ppm
Sep 28 09:03:16 dist-test-slave-286n krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:16.090610 20848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:16.091120 20848 webserver.cc:460] Webserver started at http://127.0.0.1:45657/ using document root <none> and password file <none>
I20260928 09:03:16.091384 20848 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:16.091459 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:16.091609 20848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:16.092805 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "abf3a2be381542d5abdce14786044775"
format_stamp: "Formatted at 2026-09-28 09:03:16 on dist-test-slave-286n"
server_key: "0cfc759446cb2fada012f2d768848a75"
server_key_iv: "941c6a307045c4a519f5d342d3153ced"
server_key_version: "encryptionkey@0"
I20260928 09:03:16.093200 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "abf3a2be381542d5abdce14786044775"
format_stamp: "Formatted at 2026-09-28 09:03:16 on dist-test-slave-286n"
server_key: "0cfc759446cb2fada012f2d768848a75"
server_key_iv: "941c6a307045c4a519f5d342d3153ced"
server_key_version: "encryptionkey@0"
I20260928 09:03:16.094609 20848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:16.095400 20863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:16.095605 20848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:16.095702 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "abf3a2be381542d5abdce14786044775"
format_stamp: "Formatted at 2026-09-28 09:03:16 on dist-test-slave-286n"
server_key: "0cfc759446cb2fada012f2d768848a75"
server_key_iv: "941c6a307045c4a519f5d342d3153ced"
server_key_version: "encryptionkey@0"
I20260928 09:03:16.095788 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:16.099261 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:16.101389 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:16.101557 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:16.105441 20848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42151
I20260928 09:03:16.105473 20915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42151 every 8 connection(s)
I20260928 09:03:16.105808 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:16.106971 20916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:16.108562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20848
I20260928 09:03:16.108716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:16.108902 392 external_mini_cluster.cc:1428] Setting key 26d65fbe6ce105878a38d8fd42aea05f
I20260928 09:03:16.109982 20916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775: Bootstrap starting.
I20260928 09:03:16.110781 20916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:16.111143 20916 log.cc:824] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:16.111723 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:16.109477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49529 (local address 127.0.0.1:42151)
0928 09:03:16.109674 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:16.109684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:16.109720 (+ 36us) server_negotiation.cc:407] Connection header received
0928 09:03:16.110314 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:16.110325 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:16.110368 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:16.110539 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:16.111214 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:16.111218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:16.111218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:16.111223 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:16.111264 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:16.111279 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:16.111386 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:16.111446 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 09:03:16.111507 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 09:03:16.111944 20916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775: No bootstrap required, opened a new log
I20260928 09:03:16.112771 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:16.114650 20916 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf3a2be381542d5abdce14786044775" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260928 09:03:16.114801 20916 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:16.114848 20916 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf3a2be381542d5abdce14786044775, State: Initialized, Role: FOLLOWER
I20260928 09:03:16.114974 20916 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf3a2be381542d5abdce14786044775" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260928 09:03:16.115058 20916 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:16.115093 20916 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:16.115137 20916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:16.115505 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:16.114376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39949 (local address 127.0.0.1:42151)
0928 09:03:16.114444 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:16.114446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:16.114579 (+ 133us) server_negotiation.cc:407] Connection header received
0928 09:03:16.114670 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:16.114671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:16.114676 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:16.114736 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:16.115316 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:16.115317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:16.115318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:16.115322 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:16.115336 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:16.115340 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:16.115392 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:16.115439 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 09:03:16.115450 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 09:03:16.115911 20916 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf3a2be381542d5abdce14786044775" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260928 09:03:16.116014 20916 leader_election.cc:302] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf3a2be381542d5abdce14786044775; no voters:
I20260928 09:03:16.116186 20916 leader_election.cc:288] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:16.116204 20926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:16.116358 20926 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [term 1 LEADER]: Becoming Leader. State: Replica: abf3a2be381542d5abdce14786044775, State: Running, Role: LEADER
I20260928 09:03:16.116446 20916 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:16.116443 20926 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf3a2be381542d5abdce14786044775" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260928 09:03:16.116984 20927 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf3a2be381542d5abdce14786044775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf3a2be381542d5abdce14786044775" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } } }
I20260928 09:03:16.117100 20927 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:16.117275 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf3a2be381542d5abdce14786044775. Latest consensus state: current_term: 1 leader_uuid: "abf3a2be381542d5abdce14786044775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf3a2be381542d5abdce14786044775" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } } }
I20260928 09:03:16.117314 20935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:16.117341 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:16.117820 20935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:16.119343 20935 catalog_manager.cc:1347] Generated new cluster ID: a8f97c56935a4f20a6d4c3b44d1ada89
I20260928 09:03:16.119400 20935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:16.147071 20935 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:16.147296 20935 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:16.153432 20935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf3a2be381542d5abdce14786044775: Generated new TSK 0
I20260928 09:03:16.153637 20935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:16.162379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20848
2026-09-28T09:03:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:16 dist-test-slave-286n krb5kdc[20947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:16 dist-test-slave-286n krb5kdc[20947](info): set up 2 sockets
Sep 28 09:03:16 dist-test-slave-286n krb5kdc[20947](info): commencing operation
krb5kdc: starting...
W20260928 09:03:17.193200 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:17 dist-test-slave-286n krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:17.318329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:17.398267 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:17.398468 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:17.398491 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:17.399662 20960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:17.399704 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:17.399718 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:17.399731 20960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:17.399781 20960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:17.401298 20960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:17.401546 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:17.401777 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:17.406656 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:17.406666 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:17.406687 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:17.406832 20960 server_base.cc:1034] running on GCE node
I20260928 09:03:17.407183 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:17.407465 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:17.408625 20960 hybrid_clock.cc:648] HybridClock initialized: now 1790586197408610 us; error 25 us; skew 500 ppm
Sep 28 09:03:17 dist-test-slave-286n krb5kdc[20947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:17.411300 20960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:17.411720 20960 webserver.cc:460] Webserver started at http://127.0.0.1:36621/ using document root <none> and password file <none>
I20260928 09:03:17.411962 20960 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:17.412031 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:17.412173 20960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:17.413316 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "58be330567044bd5808ae1f5cfd5ec10"
format_stamp: "Formatted at 2026-09-28 09:03:17 on dist-test-slave-286n"
server_key: "69b45f421f1700f2110f52ae7eca75ec"
server_key_iv: "8c3dfcfbe9e313bb29e69ab45db95c48"
server_key_version: "encryptionkey@0"
I20260928 09:03:17.413673 20960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "58be330567044bd5808ae1f5cfd5ec10"
format_stamp: "Formatted at 2026-09-28 09:03:17 on dist-test-slave-286n"
server_key: "69b45f421f1700f2110f52ae7eca75ec"
server_key_iv: "8c3dfcfbe9e313bb29e69ab45db95c48"
server_key_version: "encryptionkey@0"
I20260928 09:03:17.415221 20960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:17.416028 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:17.416265 20960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:17.416388 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "58be330567044bd5808ae1f5cfd5ec10"
format_stamp: "Formatted at 2026-09-28 09:03:17 on dist-test-slave-286n"
server_key: "69b45f421f1700f2110f52ae7eca75ec"
server_key_iv: "8c3dfcfbe9e313bb29e69ab45db95c48"
server_key_version: "encryptionkey@0"
I20260928 09:03:17.416462 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:17.429780 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:17.432125 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:17.432294 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:17.436416 20960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34669
I20260928 09:03:17.436450 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34669 every 8 connection(s)
I20260928 09:03:17.436820 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:17.437685 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:17.439919 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10: Bootstrap starting.
I20260928 09:03:17.440464 21028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:17.440734 21028 log.cc:824] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:17.441408 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10: No bootstrap required, opened a new log
I20260928 09:03:17.442946 21028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58be330567044bd5808ae1f5cfd5ec10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260928 09:03:17.443100 21028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:17.443125 21028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58be330567044bd5808ae1f5cfd5ec10, State: Initialized, Role: FOLLOWER
I20260928 09:03:17.443267 21028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58be330567044bd5808ae1f5cfd5ec10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260928 09:03:17.443351 21028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:17.443387 21028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:17.443440 21028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:17.443578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 20960
I20260928 09:03:17.443693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:17.443864 392 external_mini_cluster.cc:1428] Setting key 439e7568353d2ad83b25788454e05fc6
I20260928 09:03:17.444145 21028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58be330567044bd5808ae1f5cfd5ec10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260928 09:03:17.444269 21028 leader_election.cc:302] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58be330567044bd5808ae1f5cfd5ec10; no voters:
I20260928 09:03:17.444434 21028 leader_election.cc:288] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:17.444559 21032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:17.444727 21028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:17.444751 21032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [term 1 LEADER]: Becoming Leader. State: Replica: 58be330567044bd5808ae1f5cfd5ec10, State: Running, Role: LEADER
I20260928 09:03:17.444944 21032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58be330567044bd5808ae1f5cfd5ec10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } }
I20260928 09:03:17.445402 21034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58be330567044bd5808ae1f5cfd5ec10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58be330567044bd5808ae1f5cfd5ec10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } } }
I20260928 09:03:17.445501 21034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:17.445690 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58be330567044bd5808ae1f5cfd5ec10. Latest consensus state: current_term: 1 leader_uuid: "58be330567044bd5808ae1f5cfd5ec10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58be330567044bd5808ae1f5cfd5ec10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34669 } } }
I20260928 09:03:17.445760 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:17.446359 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:17.446880 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:17.448349 21041 catalog_manager.cc:1347] Generated new cluster ID: 9c5b72a1a5c046afbc7dc0f75aa21d3f
I20260928 09:03:17.448402 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:17.456459 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:17.457266 21041 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 09:03:17 dist-test-slave-286n krb5kdc[20947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:03:17.462481 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:17.444359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53343 (local address 127.0.0.1:34669)
0928 09:03:17.444586 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:17.444596 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:17.444674 (+ 78us) server_negotiation.cc:407] Connection header received
0928 09:03:17.444802 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:17.444810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:17.444869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:17.445880 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:17.447407 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:17.454981 (+ 7574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:17.455835 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:17.456458 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:17.456580 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:17.459111 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:17.459115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:17.459115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:17.459130 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:17.460895 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:17.461284 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:17.461287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:17.461287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:17.461379 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:17.461603 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:17.461604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:17.461604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:17.461830 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:17.461967 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:17.462093 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 09:03:17.462256 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
I20260928 09:03:17.463442 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:17.469295 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58be330567044bd5808ae1f5cfd5ec10: Generated new TSK 0
I20260928 09:03:17.469825 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:17.471652 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:17.464826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58225 (local address 127.0.0.1:34669)
0928 09:03:17.464870 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:17.464872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:17.465013 (+ 141us) server_negotiation.cc:407] Connection header received
0928 09:03:17.465076 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:17.465077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:17.465086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:17.465134 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:17.465652 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:17.466157 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:17.467273 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:17.467629 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:17.467673 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:17.469013 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:17.469015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:17.469016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:17.469023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:17.470622 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:17.470941 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:17.470942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:17.470943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:17.471003 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:17.471177 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:17.471178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:17.471178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:17.471304 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:17.471414 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:17.471477 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 09:03:17.471561 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:17.473670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 20960
2026-09-28T09:03:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:17 dist-test-slave-286n krb5kdc[21059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:17 dist-test-slave-286n krb5kdc[21059](info): set up 2 sockets
Sep 28 09:03:17 dist-test-slave-286n krb5kdc[21059](info): commencing operation
krb5kdc: starting...
W20260928 09:03:18.512966 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:18 dist-test-slave-286n krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:18.633702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:18.731639 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:18.731884 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:18.731925 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:18.733213 21072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:18.733278 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:18.733314 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:18.733343 21072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:18.733373 21072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:18.734856 21072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:18.735146 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:18.735387 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:18.740317 21078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:18.740310 21080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:18.740293 21077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:18.740514 21072 server_base.cc:1034] running on GCE node
I20260928 09:03:18.740782 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:18.741084 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:18.742261 21072 hybrid_clock.cc:648] HybridClock initialized: now 1790586198742247 us; error 29 us; skew 500 ppm
Sep 28 09:03:18 dist-test-slave-286n krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:18.744817 21072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:18.745306 21072 webserver.cc:460] Webserver started at http://127.0.0.1:36453/ using document root <none> and password file <none>
I20260928 09:03:18.745529 21072 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:18.745574 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:18.745723 21072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:18.746848 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "9df22c74f5db40c8833dccc548500c70"
format_stamp: "Formatted at 2026-09-28 09:03:18 on dist-test-slave-286n"
server_key: "ccad1f6c0720896e215ee6d1dc4f9221"
server_key_iv: "01d3e3f6395fbf748b53f55634b1c863"
server_key_version: "encryptionkey@0"
I20260928 09:03:18.747215 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "9df22c74f5db40c8833dccc548500c70"
format_stamp: "Formatted at 2026-09-28 09:03:18 on dist-test-slave-286n"
server_key: "ccad1f6c0720896e215ee6d1dc4f9221"
server_key_iv: "01d3e3f6395fbf748b53f55634b1c863"
server_key_version: "encryptionkey@0"
I20260928 09:03:18.748677 21072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:18.749485 21087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:18.749686 21072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:18.749791 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "9df22c74f5db40c8833dccc548500c70"
format_stamp: "Formatted at 2026-09-28 09:03:18 on dist-test-slave-286n"
server_key: "ccad1f6c0720896e215ee6d1dc4f9221"
server_key_iv: "01d3e3f6395fbf748b53f55634b1c863"
server_key_version: "encryptionkey@0"
I20260928 09:03:18.749883 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:18.769117 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:18.771152 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:18.771338 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:18.775758 21072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46153
I20260928 09:03:18.775780 21139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46153 every 8 connection(s)
I20260928 09:03:18.776152 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:18.777127 21140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:18.779402 21140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70: Bootstrap starting.
I20260928 09:03:18.779949 21140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:18.780269 21140 log.cc:824] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:18.780277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21072
I20260928 09:03:18.780395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:18.780603 392 external_mini_cluster.cc:1428] Setting key e68735462d0aa3440b74ccfbf665b80b
I20260928 09:03:18.781461 21140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70: No bootstrap required, opened a new log
I20260928 09:03:18.784276 21140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df22c74f5db40c8833dccc548500c70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46153 } }
I20260928 09:03:18.784421 21140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:18.784444 21140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df22c74f5db40c8833dccc548500c70, State: Initialized, Role: FOLLOWER
I20260928 09:03:18.784578 21140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df22c74f5db40c8833dccc548500c70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46153 } }
I20260928 09:03:18.784687 21140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:18.784727 21140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:18.784776 21140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:18.785488 21140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df22c74f5db40c8833dccc548500c70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46153 } }
I20260928 09:03:18.785601 21140 leader_election.cc:302] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df22c74f5db40c8833dccc548500c70; no voters:
I20260928 09:03:18.785759 21140 leader_election.cc:288] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:18.785843 21145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:18.786046 21145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [term 1 LEADER]: Becoming Leader. State: Replica: 9df22c74f5db40c8833dccc548500c70, State: Running, Role: LEADER
I20260928 09:03:18.786096 21140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:18.786217 21145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df22c74f5db40c8833dccc548500c70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46153 } }
I20260928 09:03:18.786789 21147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df22c74f5db40c8833dccc548500c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df22c74f5db40c8833dccc548500c70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46153 } } }
I20260928 09:03:18.786886 21147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:18.786785 21146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df22c74f5db40c8833dccc548500c70. Latest consensus state: current_term: 1 leader_uuid: "9df22c74f5db40c8833dccc548500c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df22c74f5db40c8833dccc548500c70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46153 } } }
I20260928 09:03:18.787101 21146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:18.787108 21154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:18.787631 21154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:18.788457 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:18.781193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37775 (local address 127.0.0.1:46153)
0928 09:03:18.781403 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:18.781421 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:18.781455 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:03:18.781985 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:18.781994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:18.782058 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:18.782216 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:18.782459 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:18.783920 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:18.785094 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:18.785751 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:18.785871 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:18.787608 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:18.787612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:18.787613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:18.787827 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:18.787871 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:18.787886 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:18.787966 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:18.788127 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 09:03:18.788202 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260928 09:03:18.789335 21154 catalog_manager.cc:1347] Generated new cluster ID: fb94a7c9177b4bf6b75567c83834222e
I20260928 09:03:18.789387 21154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:18.792943 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:18.797282 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:18.794296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46485 (local address 127.0.0.1:46153)
0928 09:03:18.794335 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:18.794337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:18.794455 (+ 118us) server_negotiation.cc:407] Connection header received
0928 09:03:18.794521 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:18.794522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:18.794534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:18.794581 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:18.794738 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:18.795218 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:18.795963 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:18.796168 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:18.796256 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:18.797021 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:18.797022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:18.797022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:18.797027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:18.797042 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:18.797047 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:18.797090 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:18.797155 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:03:18.797178 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":19}
I20260928 09:03:18.804826 21154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:18.805375 21154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:18.814311 21154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df22c74f5db40c8833dccc548500c70: Generated new TSK 0
I20260928 09:03:18.814550 21154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:18.817437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21072
2026-09-28T09:03:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:18 dist-test-slave-286n krb5kdc[21171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:18 dist-test-slave-286n krb5kdc[21171](info): set up 2 sockets
Sep 28 09:03:18 dist-test-slave-286n krb5kdc[21171](info): commencing operation
krb5kdc: starting...
W20260928 09:03:19.848737 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:19 dist-test-slave-286n krb5kdc[21171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:19.969239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:20.049961 21184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:20.050163 21184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:20.050187 21184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:20.051369 21184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:20.051411 21184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:20.051426 21184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:20.051440 21184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:20.051453 21184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:20.053030 21184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:20.053289 21184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:20.053560 21184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:20.058534 21190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:20.058534 21192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:20.058570 21189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:20.058676 21184 server_base.cc:1034] running on GCE node
I20260928 09:03:20.059011 21184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:20.059360 21184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:20.060534 21184 hybrid_clock.cc:648] HybridClock initialized: now 1790586200060535 us; error 43 us; skew 500 ppm
Sep 28 09:03:20 dist-test-slave-286n krb5kdc[21171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:20.063227 21184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:20.063807 21184 webserver.cc:460] Webserver started at http://127.0.0.1:45335/ using document root <none> and password file <none>
I20260928 09:03:20.064066 21184 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:20.064136 21184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:20.064275 21184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:20.065441 21184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "b0775638a0a148c29570b76dd0924cb6"
format_stamp: "Formatted at 2026-09-28 09:03:20 on dist-test-slave-286n"
server_key: "843b81d8fafbe531ec1b83e103bd08b5"
server_key_iv: "eedf10430a9ada9947bc4529a0c4f9b5"
server_key_version: "encryptionkey@0"
I20260928 09:03:20.065819 21184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0775638a0a148c29570b76dd0924cb6"
format_stamp: "Formatted at 2026-09-28 09:03:20 on dist-test-slave-286n"
server_key: "843b81d8fafbe531ec1b83e103bd08b5"
server_key_iv: "eedf10430a9ada9947bc4529a0c4f9b5"
server_key_version: "encryptionkey@0"
I20260928 09:03:20.067180 21184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:20.067920 21199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:20.068125 21184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:20.068236 21184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "b0775638a0a148c29570b76dd0924cb6"
format_stamp: "Formatted at 2026-09-28 09:03:20 on dist-test-slave-286n"
server_key: "843b81d8fafbe531ec1b83e103bd08b5"
server_key_iv: "eedf10430a9ada9947bc4529a0c4f9b5"
server_key_version: "encryptionkey@0"
I20260928 09:03:20.068316 21184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:20.071877 21184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:20.074455 21184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:20.074625 21184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:20.079123 21184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46009
I20260928 09:03:20.079154 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46009 every 8 connection(s)
I20260928 09:03:20.079519 21184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:20.080535 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:20.082854 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6: Bootstrap starting.
I20260928 09:03:20.083413 21252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:20.083644 21252 log.cc:824] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:20.084087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21184
I20260928 09:03:20.084223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:20.084400 392 external_mini_cluster.cc:1428] Setting key ae11abf2d0d1cf1bc631a9cb2997229f
I20260928 09:03:20.084420 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6: No bootstrap required, opened a new log
I20260928 09:03:20.086777 21252 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0775638a0a148c29570b76dd0924cb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 09:03:20.087033 21252 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:20.087100 21252 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0775638a0a148c29570b76dd0924cb6, State: Initialized, Role: FOLLOWER
I20260928 09:03:20.087232 21252 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0775638a0a148c29570b76dd0924cb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 09:03:20.087337 21252 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:20.087386 21252 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:20.087394 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:20.084935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40719 (local address 127.0.0.1:46009)
0928 09:03:20.085143 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:20.085154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:20.085193 (+ 39us) server_negotiation.cc:407] Connection header received
0928 09:03:20.085879 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:20.085889 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:20.085923 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:20.086086 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:20.086740 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:20.086744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:20.086744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:20.086749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:20.086793 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:20.086808 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:20.086889 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:20.086924 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 09:03:20.086964 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260928 09:03:20.087453 21252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:20.088176 21252 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0775638a0a148c29570b76dd0924cb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 09:03:20.088287 21252 leader_election.cc:302] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0775638a0a148c29570b76dd0924cb6; no voters:
I20260928 09:03:20.088441 21252 leader_election.cc:288] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:20.088490 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:20.088490 21257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:20.088650 21257 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [term 1 LEADER]: Becoming Leader. State: Replica: b0775638a0a148c29570b76dd0924cb6, State: Running, Role: LEADER
I20260928 09:03:20.088711 21252 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:20.088734 21257 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0775638a0a148c29570b76dd0924cb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } }
I20260928 09:03:20.089047 21257 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0775638a0a148c29570b76dd0924cb6. Latest consensus state: current_term: 1 leader_uuid: "b0775638a0a148c29570b76dd0924cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0775638a0a148c29570b76dd0924cb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } } }
I20260928 09:03:20.089136 21257 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:20.089058 21258 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0775638a0a148c29570b76dd0924cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0775638a0a148c29570b76dd0924cb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46009 } } }
I20260928 09:03:20.089394 21258 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6 [sys.catalog]: This master's current role is: LEADER
W20260928 09:03:20.090546 21276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:20.090605 21276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:20.090647 21263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:20.090816 21263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:20.091710 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:20.090255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57163 (local address 127.0.0.1:46009)
0928 09:03:20.090283 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:20.090284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:20.090510 (+ 226us) server_negotiation.cc:407] Connection header received
0928 09:03:20.090673 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:20.090674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:20.090680 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:20.090730 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:20.091462 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:20.091463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:20.091464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:20.091469 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:20.091484 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:20.091488 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:20.091555 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:20.091641 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 09:03:20.091653 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260928 09:03:20.092509 21263 catalog_manager.cc:1347] Generated new cluster ID: 68cb79f5c35342bca9dfde8443df189d
I20260928 09:03:20.092561 21263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:20.102828 21263 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:20.103015 21263 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:20.108593 21263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0775638a0a148c29570b76dd0924cb6: Generated new TSK 0
I20260928 09:03:20.108816 21263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:20.118396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21184
2026-09-28T09:03:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:20 dist-test-slave-286n krb5kdc[21283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:20 dist-test-slave-286n krb5kdc[21283](info): set up 2 sockets
Sep 28 09:03:20 dist-test-slave-286n krb5kdc[21283](info): commencing operation
krb5kdc: starting...
W20260928 09:03:21.176798 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:21 dist-test-slave-286n krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:21.297784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:21.380228 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:21.380443 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:21.380466 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:21.381579 21296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:21.381616 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:21.381631 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:21.381644 21296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:21.381656 21296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:21.383067 21296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:21.383260 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:21.383437 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:03:21.388365 21296 server_base.cc:1034] running on GCE node
W20260928 09:03:21.388406 21302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:21.388386 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:21.388363 21301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:21.388857 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:21.389110 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:21.390280 21296 hybrid_clock.cc:648] HybridClock initialized: now 1790586201390271 us; error 38 us; skew 500 ppm
Sep 28 09:03:21 dist-test-slave-286n krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:21.397609 21296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:21.398133 21296 webserver.cc:460] Webserver started at http://127.0.0.1:38349/ using document root <none> and password file <none>
I20260928 09:03:21.398380 21296 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:21.398450 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:21.398610 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:21.399755 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "ba4b56070fa84bd992b515e687d346dc"
format_stamp: "Formatted at 2026-09-28 09:03:21 on dist-test-slave-286n"
server_key: "745b347f228c4713585a1a1c0b05908b"
server_key_iv: "69f36569d28db6dbe663345aecf1097b"
server_key_version: "encryptionkey@0"
I20260928 09:03:21.400144 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba4b56070fa84bd992b515e687d346dc"
format_stamp: "Formatted at 2026-09-28 09:03:21 on dist-test-slave-286n"
server_key: "745b347f228c4713585a1a1c0b05908b"
server_key_iv: "69f36569d28db6dbe663345aecf1097b"
server_key_version: "encryptionkey@0"
I20260928 09:03:21.401635 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:21.402444 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:21.402642 21296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:21.402732 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "ba4b56070fa84bd992b515e687d346dc"
format_stamp: "Formatted at 2026-09-28 09:03:21 on dist-test-slave-286n"
server_key: "745b347f228c4713585a1a1c0b05908b"
server_key_iv: "69f36569d28db6dbe663345aecf1097b"
server_key_version: "encryptionkey@0"
I20260928 09:03:21.402791 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:21.415408 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:21.417840 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:21.417960 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:21.422276 21296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44605
I20260928 09:03:21.422308 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44605 every 8 connection(s)
I20260928 09:03:21.422686 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:21.423586 21364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:21.425976 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc: Bootstrap starting.
I20260928 09:03:21.426569 21364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:21.426855 21364 log.cc:824] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:21.427569 21364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc: No bootstrap required, opened a new log
I20260928 09:03:21.429226 21364 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b56070fa84bd992b515e687d346dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44605 } }
I20260928 09:03:21.429397 21364 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:21.429448 21364 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4b56070fa84bd992b515e687d346dc, State: Initialized, Role: FOLLOWER
I20260928 09:03:21.429620 21364 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b56070fa84bd992b515e687d346dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44605 } }
I20260928 09:03:21.429700 21364 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:21.429744 21364 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:21.429795 21364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:21.430335 21364 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b56070fa84bd992b515e687d346dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44605 } }
I20260928 09:03:21.430454 21364 leader_election.cc:302] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba4b56070fa84bd992b515e687d346dc; no voters:
I20260928 09:03:21.430636 21364 leader_election.cc:288] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:21.430747 21367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:21.430896 21364 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:21.430949 21367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [term 1 LEADER]: Becoming Leader. State: Replica: ba4b56070fa84bd992b515e687d346dc, State: Running, Role: LEADER
I20260928 09:03:21.431041 21367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b56070fa84bd992b515e687d346dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44605 } }
I20260928 09:03:21.431695 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba4b56070fa84bd992b515e687d346dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b56070fa84bd992b515e687d346dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44605 } } }
I20260928 09:03:21.431705 21369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4b56070fa84bd992b515e687d346dc. Latest consensus state: current_term: 1 leader_uuid: "ba4b56070fa84bd992b515e687d346dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba4b56070fa84bd992b515e687d346dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44605 } } }
I20260928 09:03:21.431792 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:21.431809 21369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:21.432061 21376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:21.432559 21376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:21.432832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21296
I20260928 09:03:21.432955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:21.433152 392 external_mini_cluster.cc:1428] Setting key 5e711e5508a66d3972703036212fbaa1
I20260928 09:03:21.437156 21376 catalog_manager.cc:1347] Generated new cluster ID: 636676835bfa434597a47d92df0b31fd
I20260928 09:03:21.437237 21376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:21 dist-test-slave-286n krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:03:21.444787 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:21.433828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:59411 (local address 127.0.0.1:44605)
0928 09:03:21.433999 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:21.434009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:21.434039 (+ 30us) server_negotiation.cc:407] Connection header received
0928 09:03:21.434168 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:21.434173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:21.434218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:21.434329 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:21.435088 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:21.436095 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:21.437287 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:21.437895 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:21.438007 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:21.441002 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:21.441007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:21.441007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:21.441020 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:21.443055 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:21.443491 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:21.443494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:21.443495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:21.443604 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:21.443829 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:21.443830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:21.443830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:21.444164 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:21.444328 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:21.444410 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 09:03:21.444515 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260928 09:03:21.445870 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:21.454550 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:21.447298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:47977 (local address 127.0.0.1:44605)
0928 09:03:21.447331 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:21.447332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:21.447451 (+ 119us) server_negotiation.cc:407] Connection header received
0928 09:03:21.447494 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:21.447494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:21.447514 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:21.447571 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:21.448178 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:21.448960 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:21.449975 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:21.450274 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:21.450326 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:21.451711 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:21.451712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:21.451713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:21.451719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:21.453309 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:21.453715 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:21.453716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:21.453716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:21.453770 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:21.453998 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:21.453999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:21.453999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:21.454118 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:21.454212 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:21.454369 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 09:03:21.454433 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:03:21.455615 21376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:21.456317 21376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:21.465332 21376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4b56070fa84bd992b515e687d346dc: Generated new TSK 0
I20260928 09:03:21.465545 21376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:21.468364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21296
W20260928 09:03:21.472898 21281 connection.cc:535] client connection to 127.0.0.1:44605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44605): Connection reset by peer (error 104)
2026-09-28T09:03:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:21 dist-test-slave-286n krb5kdc[21395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:21 dist-test-slave-286n krb5kdc[21395](info): set up 2 sockets
Sep 28 09:03:21 dist-test-slave-286n krb5kdc[21395](info): commencing operation
krb5kdc: starting...
W20260928 09:03:22.513031 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:22 dist-test-slave-286n krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:22.632906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:22.717708 21408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:22.717957 21408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:22.717998 21408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:22.719203 21408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:22.719269 21408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:22.719295 21408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:22.719326 21408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:22.719367 21408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:22.720922 21408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:22.721187 21408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:22.721418 21408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:22.726459 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:22.726467 21408 server_base.cc:1034] running on GCE node
W20260928 09:03:22.726508 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:22.726626 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:22.726946 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:22.727278 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:22.728443 21408 hybrid_clock.cc:648] HybridClock initialized: now 1790586202728427 us; error 38 us; skew 500 ppm
Sep 28 09:03:22 dist-test-slave-286n krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:22.730926 21408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:22.731382 21408 webserver.cc:460] Webserver started at http://127.0.0.1:37945/ using document root <none> and password file <none>
I20260928 09:03:22.731622 21408 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:22.731667 21408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:22.731815 21408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:22.732939 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "dae5854a67b643f9abf6beec73af1d69"
format_stamp: "Formatted at 2026-09-28 09:03:22 on dist-test-slave-286n"
server_key: "f6fb732a76d94d7d6a543cb0fdc1aa2b"
server_key_iv: "f95b87a91ee2c9c831c5f613dd70d79c"
server_key_version: "encryptionkey@0"
I20260928 09:03:22.733284 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "dae5854a67b643f9abf6beec73af1d69"
format_stamp: "Formatted at 2026-09-28 09:03:22 on dist-test-slave-286n"
server_key: "f6fb732a76d94d7d6a543cb0fdc1aa2b"
server_key_iv: "f95b87a91ee2c9c831c5f613dd70d79c"
server_key_version: "encryptionkey@0"
I20260928 09:03:22.734620 21408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:22.735373 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:22.735674 21408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:22.735759 21408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "dae5854a67b643f9abf6beec73af1d69"
format_stamp: "Formatted at 2026-09-28 09:03:22 on dist-test-slave-286n"
server_key: "f6fb732a76d94d7d6a543cb0fdc1aa2b"
server_key_iv: "f95b87a91ee2c9c831c5f613dd70d79c"
server_key_version: "encryptionkey@0"
I20260928 09:03:22.735811 21408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:22.746277 21408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:22.748389 21408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:22.748534 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:22.752400 21408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38445
I20260928 09:03:22.752497 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38445 every 8 connection(s)
I20260928 09:03:22.752864 21408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:22.753906 21476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:22.756184 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69: Bootstrap starting.
I20260928 09:03:22.756831 21476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:22.757113 21476 log.cc:824] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:22.757777 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69: No bootstrap required, opened a new log
I20260928 09:03:22.757807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21408
I20260928 09:03:22.757920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:22.758144 392 external_mini_cluster.cc:1428] Setting key dcd159005cf36757407e169ad7eb8001
I20260928 09:03:22.760095 21476 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae5854a67b643f9abf6beec73af1d69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260928 09:03:22.760303 21476 raft_consensus.cc:374] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:22.760345 21476 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae5854a67b643f9abf6beec73af1d69, State: Initialized, Role: FOLLOWER
I20260928 09:03:22.760483 21476 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae5854a67b643f9abf6beec73af1d69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260928 09:03:22.760557 21476 raft_consensus.cc:388] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:22.760587 21476 raft_consensus.cc:482] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:22.760654 21476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:22.761327 21476 raft_consensus.cc:504] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae5854a67b643f9abf6beec73af1d69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260928 09:03:22.761433 21476 leader_election.cc:302] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dae5854a67b643f9abf6beec73af1d69; no voters:
I20260928 09:03:22.761575 21476 leader_election.cc:288] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:22.761622 21481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:22.761783 21481 raft_consensus.cc:686] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [term 1 LEADER]: Becoming Leader. State: Replica: dae5854a67b643f9abf6beec73af1d69, State: Running, Role: LEADER
I20260928 09:03:22.761839 21476 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:22.761969 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae5854a67b643f9abf6beec73af1d69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260928 09:03:22.762290 21481 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae5854a67b643f9abf6beec73af1d69. Latest consensus state: current_term: 1 leader_uuid: "dae5854a67b643f9abf6beec73af1d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae5854a67b643f9abf6beec73af1d69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } } }
I20260928 09:03:22.762416 21481 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:22.762647 21490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:22.763154 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dae5854a67b643f9abf6beec73af1d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae5854a67b643f9abf6beec73af1d69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } } }
I20260928 09:03:22.763262 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:22.763439 21490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:22.765303 21490 catalog_manager.cc:1347] Generated new cluster ID: c20b46b9bf6c43678bd6de7b15b9323c
I20260928 09:03:22.765381 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:22.758672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:53323 (local address 127.0.0.1:38445)
0928 09:03:22.758867 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:22.758877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:22.758911 (+ 34us) server_negotiation.cc:407] Connection header received
0928 09:03:22.759643 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:22.759652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:22.759724 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:22.759894 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:22.760034 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:22.761749 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:22.762932 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:22.763631 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:22.763741 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:22.764737 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:22.764741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:22.764741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:22.764906 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:22.764944 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:22.764971 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:22.765065 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:22.765177 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 09:03:22.765205 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260928 09:03:22.765372 21490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:22.766300 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:22.772254 21490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:22.773026 21490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:22.774904 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:22.768198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:43547 (local address 127.0.0.1:38445)
0928 09:03:22.768237 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:22.768240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:22.771963 (+ 3723us) server_negotiation.cc:407] Connection header received
0928 09:03:22.772063 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:22.772064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:22.772075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:22.772138 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:22.772313 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:22.772919 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:22.773691 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:22.773889 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:22.773962 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:22.774641 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:22.774643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:22.774643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:22.774648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:22.774662 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:22.774667 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:22.774735 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:22.774800 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:03:22.774809 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:22.777089 21490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dae5854a67b643f9abf6beec73af1d69: Generated new TSK 0
I20260928 09:03:22.777289 21490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:22.780771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21408
W20260928 09:03:22.784775 21393 connection.cc:535] client connection to 127.0.0.1:38445 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38445): Connection reset by peer (error 104)
2026-09-28T09:03:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:22 dist-test-slave-286n krb5kdc[21507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:22 dist-test-slave-286n krb5kdc[21507](info): set up 2 sockets
Sep 28 09:03:22 dist-test-slave-286n krb5kdc[21507](info): commencing operation
krb5kdc: starting...
W20260928 09:03:23.816876 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:23 dist-test-slave-286n krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:23.936103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:24.016417 21520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:24.016660 21520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:24.016685 21520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:24.017874 21520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:24.017928 21520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:24.017944 21520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:24.017957 21520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:24.017969 21520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:24.019404 21520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:24.019594 21520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:24.019779 21520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:24.024529 21526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:24.024523 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:24.024797 21520 server_base.cc:1034] running on GCE node
W20260928 09:03:24.024524 21528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:24.025079 21520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:24.025393 21520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:24.026552 21520 hybrid_clock.cc:648] HybridClock initialized: now 1790586204026534 us; error 40 us; skew 500 ppm
Sep 28 09:03:24 dist-test-slave-286n krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:24.029558 21520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:24.030097 21520 webserver.cc:460] Webserver started at http://127.0.0.1:36293/ using document root <none> and password file <none>
I20260928 09:03:24.030349 21520 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:24.030406 21520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:24.030535 21520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:24.031623 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "bda0013ae73f406591e1995d2d05a8aa"
format_stamp: "Formatted at 2026-09-28 09:03:24 on dist-test-slave-286n"
server_key: "d0f5c3a0cd278e4c934f16294dc04ad5"
server_key_iv: "dfbb09753ae05dd8785540be0923f988"
server_key_version: "encryptionkey@0"
I20260928 09:03:24.031988 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "bda0013ae73f406591e1995d2d05a8aa"
format_stamp: "Formatted at 2026-09-28 09:03:24 on dist-test-slave-286n"
server_key: "d0f5c3a0cd278e4c934f16294dc04ad5"
server_key_iv: "dfbb09753ae05dd8785540be0923f988"
server_key_version: "encryptionkey@0"
I20260928 09:03:24.033401 21520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:24.034227 21535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:24.034418 21520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:24.034519 21520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "bda0013ae73f406591e1995d2d05a8aa"
format_stamp: "Formatted at 2026-09-28 09:03:24 on dist-test-slave-286n"
server_key: "d0f5c3a0cd278e4c934f16294dc04ad5"
server_key_iv: "dfbb09753ae05dd8785540be0923f988"
server_key_version: "encryptionkey@0"
I20260928 09:03:24.034591 21520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:24.037995 21520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:24.040513 21520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:24.040738 21520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:24.044950 21520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39691
I20260928 09:03:24.045006 21587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39691 every 8 connection(s)
I20260928 09:03:24.045353 21520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:24.046379 21588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:24.048734 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa: Bootstrap starting.
I20260928 09:03:24.049292 21588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:24.049553 21588 log.cc:824] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:24.050228 21588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa: No bootstrap required, opened a new log
I20260928 09:03:24.050997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21520
I20260928 09:03:24.051122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:24.051335 392 external_mini_cluster.cc:1428] Setting key fadfe98ae70da466b9653c0367ea60ff
I20260928 09:03:24.052811 21588 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0013ae73f406591e1995d2d05a8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260928 09:03:24.052958 21588 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:24.052981 21588 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda0013ae73f406591e1995d2d05a8aa, State: Initialized, Role: FOLLOWER
I20260928 09:03:24.053107 21588 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0013ae73f406591e1995d2d05a8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260928 09:03:24.053217 21588 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:24.053308 21588 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:24.053390 21588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:24.054168 21588 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0013ae73f406591e1995d2d05a8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260928 09:03:24.054314 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:24.051899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:54671 (local address 127.0.0.1:39691)
0928 09:03:24.052074 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:24.052081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:24.052109 (+ 28us) server_negotiation.cc:407] Connection header received
0928 09:03:24.052861 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:24.052869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:24.052899 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:24.053089 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:24.053804 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:24.053807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:24.053807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:24.053811 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:24.053853 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:24.053864 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:24.053924 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:24.054010 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 09:03:24.054063 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 09:03:24.054309 21588 leader_election.cc:302] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda0013ae73f406591e1995d2d05a8aa; no voters:
I20260928 09:03:24.054536 21588 leader_election.cc:288] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:24.054564 21593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:24.054689 21593 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [term 1 LEADER]: Becoming Leader. State: Replica: bda0013ae73f406591e1995d2d05a8aa, State: Running, Role: LEADER
I20260928 09:03:24.054802 21588 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:24.054824 21593 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0013ae73f406591e1995d2d05a8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } }
I20260928 09:03:24.055114 21593 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda0013ae73f406591e1995d2d05a8aa. Latest consensus state: current_term: 1 leader_uuid: "bda0013ae73f406591e1995d2d05a8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0013ae73f406591e1995d2d05a8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } } }
I20260928 09:03:24.055197 21594 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda0013ae73f406591e1995d2d05a8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda0013ae73f406591e1995d2d05a8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39691 } } }
I20260928 09:03:24.055254 21593 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:24.055469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:24.055532 21594 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:24.055658 21601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:24.056180 21601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:24.058488 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:24.057449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:52121 (local address 127.0.0.1:39691)
0928 09:03:24.057502 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:24.057504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:24.057512 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:24.057532 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:24.057533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:24.057555 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:24.057631 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:24.058252 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:24.058254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:24.058254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:24.058259 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:24.058273 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:24.058278 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:24.058339 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:24.058403 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 09:03:24.058419 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:03:24.061244 21601 catalog_manager.cc:1347] Generated new cluster ID: 59f135ac6bd74c2ca269b8a4f00ad529
I20260928 09:03:24.061328 21601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:24.073618 21601 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:24.073827 21601 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:24.080977 21601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda0013ae73f406591e1995d2d05a8aa: Generated new TSK 0
I20260928 09:03:24.081192 21601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:24.087247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21520
2026-09-28T09:03:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:24 dist-test-slave-286n krb5kdc[21619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:24 dist-test-slave-286n krb5kdc[21619](info): set up 2 sockets
Sep 28 09:03:24 dist-test-slave-286n krb5kdc[21619](info): commencing operation
krb5kdc: starting...
W20260928 09:03:25.136718 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:25 dist-test-slave-286n krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:25.256330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:25.336716 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:25.336967 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:25.337010 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:25.338204 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:25.338276 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:25.338300 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:25.338330 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:25.338361 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:25.339886 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:25.340175 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:25.340406 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:25.345288 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:25.345288 21637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:25.345314 21638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:25.345434 21632 server_base.cc:1034] running on GCE node
I20260928 09:03:25.345822 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:25.346110 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:25.347295 21632 hybrid_clock.cc:648] HybridClock initialized: now 1790586205347296 us; error 42 us; skew 500 ppm
Sep 28 09:03:25 dist-test-slave-286n krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:25.349709 21632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:25.350149 21632 webserver.cc:460] Webserver started at http://127.0.0.1:36425/ using document root <none> and password file <none>
I20260928 09:03:25.350374 21632 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:25.350440 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:25.350584 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:25.351708 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "0e123197d38a4e37b9c171002d6a21e8"
format_stamp: "Formatted at 2026-09-28 09:03:25 on dist-test-slave-286n"
server_key: "27cbe6f2c2ba319622765aaa9c5d712b"
server_key_iv: "f92f2616d9dea2b8de8d4963cd3ff065"
server_key_version: "encryptionkey@0"
I20260928 09:03:25.352070 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e123197d38a4e37b9c171002d6a21e8"
format_stamp: "Formatted at 2026-09-28 09:03:25 on dist-test-slave-286n"
server_key: "27cbe6f2c2ba319622765aaa9c5d712b"
server_key_iv: "f92f2616d9dea2b8de8d4963cd3ff065"
server_key_version: "encryptionkey@0"
I20260928 09:03:25.353466 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:25.354161 21647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:25.354323 21632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:25.354420 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "0e123197d38a4e37b9c171002d6a21e8"
format_stamp: "Formatted at 2026-09-28 09:03:25 on dist-test-slave-286n"
server_key: "27cbe6f2c2ba319622765aaa9c5d712b"
server_key_iv: "f92f2616d9dea2b8de8d4963cd3ff065"
server_key_version: "encryptionkey@0"
I20260928 09:03:25.354496 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:25.378376 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:25.380290 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:25.380455 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:25.384395 21632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41949
I20260928 09:03:25.384440 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41949 every 8 connection(s)
I20260928 09:03:25.384809 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:25.385670 21700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:25.387950 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8: Bootstrap starting.
I20260928 09:03:25.388521 21700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:25.388839 21700 log.cc:824] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:25.389510 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8: No bootstrap required, opened a new log
I20260928 09:03:25.391124 21700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e123197d38a4e37b9c171002d6a21e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41949 } }
I20260928 09:03:25.391258 21700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:25.391279 21700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e123197d38a4e37b9c171002d6a21e8, State: Initialized, Role: FOLLOWER
I20260928 09:03:25.391451 21700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e123197d38a4e37b9c171002d6a21e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41949 } }
I20260928 09:03:25.391482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21632
I20260928 09:03:25.391568 21700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:25.391608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:25.391614 21700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:25.391664 21700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:25.391827 392 external_mini_cluster.cc:1428] Setting key 0de1ccd8e8901bbc085c7080b6775b01
I20260928 09:03:25.392362 21700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e123197d38a4e37b9c171002d6a21e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41949 } }
I20260928 09:03:25.392515 21700 leader_election.cc:302] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e123197d38a4e37b9c171002d6a21e8; no voters:
I20260928 09:03:25.392745 21700 leader_election.cc:288] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:25.392767 21705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:25.392879 21705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [term 1 LEADER]: Becoming Leader. State: Replica: 0e123197d38a4e37b9c171002d6a21e8, State: Running, Role: LEADER
I20260928 09:03:25.393049 21700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:25.393013 21705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e123197d38a4e37b9c171002d6a21e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41949 } }
I20260928 09:03:25.393628 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e123197d38a4e37b9c171002d6a21e8. Latest consensus state: current_term: 1 leader_uuid: "0e123197d38a4e37b9c171002d6a21e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e123197d38a4e37b9c171002d6a21e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41949 } } }
I20260928 09:03:25.393640 21706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e123197d38a4e37b9c171002d6a21e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e123197d38a4e37b9c171002d6a21e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41949 } } }
I20260928 09:03:25.393795 21706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:25.393781 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:25.394409 21712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:25.394932 21712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:25.396616 21712 catalog_manager.cc:1347] Generated new cluster ID: 2afa4b0a35df4068a4d1607f2106acbf
I20260928 09:03:25.396685 21712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:25 dist-test-slave-286n krb5kdc[21619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 09:03:25.403481 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:25.392314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:35833 (local address 127.0.0.1:41949)
0928 09:03:25.392470 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:25.392480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:25.392524 (+ 44us) server_negotiation.cc:407] Connection header received
0928 09:03:25.392671 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:25.392678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:25.392739 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:25.393628 (+ 889us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:25.393731 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:25.395238 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:25.396177 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:25.396810 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:25.396917 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:25.399771 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:25.399775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:25.399775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:25.399787 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:25.401776 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:25.402225 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:25.402228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:25.402228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:25.402341 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:25.402602 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:25.402603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:25.402603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:25.402888 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:25.403024 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:25.403179 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 09:03:25.403273 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260928 09:03:25.404496 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:25.413070 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:25.405858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:59147 (local address 127.0.0.1:41949)
0928 09:03:25.405913 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:25.405915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:25.406132 (+ 217us) server_negotiation.cc:407] Connection header received
0928 09:03:25.406177 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:25.406178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:25.406189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:25.406253 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:25.406824 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:25.407594 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:25.408543 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:25.408915 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:25.408987 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:25.410274 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:25.410275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:25.410276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:25.410283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:25.411830 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:25.412234 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:25.412235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:25.412235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:25.412292 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:25.412519 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:25.412519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:25.412520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:25.412682 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:25.412787 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:25.412913 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 09:03:25.412975 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 09:03:25.413517 21712 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:25.413998 21712 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:25.427706 21712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e123197d38a4e37b9c171002d6a21e8: Generated new TSK 0
I20260928 09:03:25.427927 21712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:25.431305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21632
W20260928 09:03:25.435585 21617 connection.cc:535] client connection to 127.0.0.1:41949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41949): Connection reset by peer (error 104)
2026-09-28T09:03:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:25 dist-test-slave-286n krb5kdc[21731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:25 dist-test-slave-286n krb5kdc[21731](info): set up 2 sockets
Sep 28 09:03:25 dist-test-slave-286n krb5kdc[21731](info): commencing operation
krb5kdc: starting...
W20260928 09:03:26.460901 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:26 dist-test-slave-286n krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:26.579674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:26.659786 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:26.659998 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:26.660022 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:26.661283 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:26.661329 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:26.661343 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:26.661356 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:26.661375 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:26.662868 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:26.663129 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:26.663358 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:26.668380 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:26.668387 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:26.668540 21744 server_base.cc:1034] running on GCE node
W20260928 09:03:26.668387 21752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:26.668828 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:26.669164 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:26.670334 21744 hybrid_clock.cc:648] HybridClock initialized: now 1790586206670295 us; error 51 us; skew 500 ppm
Sep 28 09:03:26 dist-test-slave-286n krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:26.678921 21744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:26.679486 21744 webserver.cc:460] Webserver started at http://127.0.0.1:45257/ using document root <none> and password file <none>
I20260928 09:03:26.679726 21744 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:26.679791 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:26.679934 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:26.681097 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "9c9b41e45b194c228b1cb48b690be743"
format_stamp: "Formatted at 2026-09-28 09:03:26 on dist-test-slave-286n"
server_key: "8e5fb79d788d33dd757e395f0472da59"
server_key_iv: "07708248105e5b78f938361bd5e8330e"
server_key_version: "encryptionkey@0"
I20260928 09:03:26.681465 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c9b41e45b194c228b1cb48b690be743"
format_stamp: "Formatted at 2026-09-28 09:03:26 on dist-test-slave-286n"
server_key: "8e5fb79d788d33dd757e395f0472da59"
server_key_iv: "07708248105e5b78f938361bd5e8330e"
server_key_version: "encryptionkey@0"
I20260928 09:03:26.682904 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:26.683693 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:26.683941 21744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:26.684062 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "9c9b41e45b194c228b1cb48b690be743"
format_stamp: "Formatted at 2026-09-28 09:03:26 on dist-test-slave-286n"
server_key: "8e5fb79d788d33dd757e395f0472da59"
server_key_iv: "07708248105e5b78f938361bd5e8330e"
server_key_version: "encryptionkey@0"
I20260928 09:03:26.684144 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:26.705740 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:26.707769 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:26.707937 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:26.712247 21744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37799
I20260928 09:03:26.712301 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37799 every 8 connection(s)
I20260928 09:03:26.712703 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:26.713687 21812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:26.714562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21744
I20260928 09:03:26.714676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:26.714870 392 external_mini_cluster.cc:1428] Setting key a4759db752a719f75f5413752e58f073
I20260928 09:03:26.716287 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743: Bootstrap starting.
I20260928 09:03:26.717046 21812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:26.717398 21812 log.cc:824] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:26.718256 21812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743: No bootstrap required, opened a new log
I20260928 09:03:26.720562 21812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9b41e45b194c228b1cb48b690be743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37799 } }
I20260928 09:03:26.720778 21812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:26.720839 21812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9b41e45b194c228b1cb48b690be743, State: Initialized, Role: FOLLOWER
I20260928 09:03:26.720978 21812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9b41e45b194c228b1cb48b690be743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37799 } }
I20260928 09:03:26.720995 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:26.715384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:57491 (local address 127.0.0.1:37799)
0928 09:03:26.715562 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:26.715570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:26.715595 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:03:26.716068 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:26.716074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:26.716124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:26.716241 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:26.716478 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:26.717825 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:26.718949 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:26.719365 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:26.719460 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:26.720196 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:26.720200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:26.720200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:26.720366 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:26.720411 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:26.720423 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:26.720498 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:26.720626 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 09:03:26.720698 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 09:03:26.721065 21812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:26.721107 21812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:26.721151 21812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:26.721714 21812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9b41e45b194c228b1cb48b690be743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37799 } }
I20260928 09:03:26.721830 21812 leader_election.cc:302] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c9b41e45b194c228b1cb48b690be743; no voters:
I20260928 09:03:26.721961 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:26.722020 21812 leader_election.cc:288] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:26.722064 21817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:26.722209 21817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [term 1 LEADER]: Becoming Leader. State: Replica: 9c9b41e45b194c228b1cb48b690be743, State: Running, Role: LEADER
I20260928 09:03:26.722285 21812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:26.722337 21817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9b41e45b194c228b1cb48b690be743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37799 } }
I20260928 09:03:26.722711 21817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c9b41e45b194c228b1cb48b690be743. Latest consensus state: current_term: 1 leader_uuid: "9c9b41e45b194c228b1cb48b690be743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9b41e45b194c228b1cb48b690be743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37799 } } }
I20260928 09:03:26.722720 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c9b41e45b194c228b1cb48b690be743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9b41e45b194c228b1cb48b690be743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37799 } } }
I20260928 09:03:26.722860 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:26.723058 21817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:26.723219 21824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:26.723742 21824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:26.725898 21824 catalog_manager.cc:1347] Generated new cluster ID: 18b038ea7ba143989827e45c0fef6bfe
I20260928 09:03:26.725952 21824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:26.727793 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:26.724577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:55955 (local address 127.0.0.1:37799)
0928 09:03:26.724616 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:26.724618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:26.724741 (+ 123us) server_negotiation.cc:407] Connection header received
0928 09:03:26.724850 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:26.724850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:26.724861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:26.724895 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:26.725090 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:26.725732 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:26.726604 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:26.726818 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:26.726886 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:26.727572 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:26.727574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:26.727574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:26.727578 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:26.727592 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:26.727596 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:26.727650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:26.727710 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 09:03:26.727718 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:26.737557 21824 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:26.738073 21824 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:26.747438 21824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c9b41e45b194c228b1cb48b690be743: Generated new TSK 0
I20260928 09:03:26.747633 21824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:26.753162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21744
W20260928 09:03:26.757319 21729 connection.cc:535] client connection to 127.0.0.1:37799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37799): Connection reset by peer (error 104)
2026-09-28T09:03:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:26 dist-test-slave-286n krb5kdc[21843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:26 dist-test-slave-286n krb5kdc[21843](info): set up 2 sockets
Sep 28 09:03:26 dist-test-slave-286n krb5kdc[21843](info): commencing operation
krb5kdc: starting...
W20260928 09:03:27.788928 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:27 dist-test-slave-286n krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 09:03:27.911605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:27.992375 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:27.992575 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:27.992599 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:27.993749 21856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:27.993793 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:27.993806 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:27.993819 21856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:27.993870 21856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:27.995287 21856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:27.995529 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:27.995757 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:03:28.000739 21856 server_base.cc:1034] running on GCE node
W20260928 09:03:28.000777 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:28.000751 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:28.000898 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:28.001148 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:28.001462 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:28.002612 21856 hybrid_clock.cc:648] HybridClock initialized: now 1790586208002597 us; error 33 us; skew 500 ppm
Sep 28 09:03:28 dist-test-slave-286n krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:28.005092 21856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 09:03:28.005544 21856 webserver.cc:460] Webserver started at http://127.0.0.1:45481/ using document root <none> and password file <none>
I20260928 09:03:28.005765 21856 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:28.005810 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:28.005945 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:28.006979 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "b4f2d4a296d841029029ac2f527464da"
format_stamp: "Formatted at 2026-09-28 09:03:28 on dist-test-slave-286n"
server_key: "3baf3afa6fc910cc57e68538c2b1efdb"
server_key_iv: "08ef64339741f7ba43c2fa0c557f4edb"
server_key_version: "encryptionkey@0"
I20260928 09:03:28.007331 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4f2d4a296d841029029ac2f527464da"
format_stamp: "Formatted at 2026-09-28 09:03:28 on dist-test-slave-286n"
server_key: "3baf3afa6fc910cc57e68538c2b1efdb"
server_key_iv: "08ef64339741f7ba43c2fa0c557f4edb"
server_key_version: "encryptionkey@0"
I20260928 09:03:28.008625 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 09:03:28.009445 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:28.009644 21856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:28.009734 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "b4f2d4a296d841029029ac2f527464da"
format_stamp: "Formatted at 2026-09-28 09:03:28 on dist-test-slave-286n"
server_key: "3baf3afa6fc910cc57e68538c2b1efdb"
server_key_iv: "08ef64339741f7ba43c2fa0c557f4edb"
server_key_version: "encryptionkey@0"
I20260928 09:03:28.009809 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:28.013309 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:28.015200 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:28.015339 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:28.019476 21856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44391
I20260928 09:03:28.019544 21923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44391 every 8 connection(s)
I20260928 09:03:28.019826 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:28.020834 21924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:28.023206 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da: Bootstrap starting.
I20260928 09:03:28.023768 21924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:28.024024 21924 log.cc:824] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:28.024798 21924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da: No bootstrap required, opened a new log
I20260928 09:03:28.026475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21856
I20260928 09:03:28.026605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:28.026501 21924 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2d4a296d841029029ac2f527464da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260928 09:03:28.026671 21924 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:28.026722 21924 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f2d4a296d841029029ac2f527464da, State: Initialized, Role: FOLLOWER
I20260928 09:03:28.026823 392 external_mini_cluster.cc:1428] Setting key 118510d045e33ae67dccaf12e89bc5f1
I20260928 09:03:28.026889 21924 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2d4a296d841029029ac2f527464da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260928 09:03:28.026980 21924 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:28.027019 21924 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:28.027055 21924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:28.027804 21924 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2d4a296d841029029ac2f527464da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260928 09:03:28.027922 21924 leader_election.cc:302] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f2d4a296d841029029ac2f527464da; no voters:
I20260928 09:03:28.028081 21924 leader_election.cc:288] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:28.028108 21929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:28.028264 21929 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [term 1 LEADER]: Becoming Leader. State: Replica: b4f2d4a296d841029029ac2f527464da, State: Running, Role: LEADER
I20260928 09:03:28.028335 21924 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:28.028453 21929 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2d4a296d841029029ac2f527464da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260928 09:03:28.029040 21931 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f2d4a296d841029029ac2f527464da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2d4a296d841029029ac2f527464da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } } }
I20260928 09:03:28.029408 21931 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:28.029052 21930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f2d4a296d841029029ac2f527464da. Latest consensus state: current_term: 1 leader_uuid: "b4f2d4a296d841029029ac2f527464da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f2d4a296d841029029ac2f527464da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } } }
I20260928 09:03:28.029678 21930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:28.029893 21941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:28.029930 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:28.027347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:56281 (local address 127.0.0.1:44391)
0928 09:03:28.027858 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:28.027868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:28.027905 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:03:28.028024 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:28.028032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:28.028065 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:28.028272 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:28.029225 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:28.029228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:28.029228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:28.029233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:28.029288 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:28.029304 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:28.029378 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:28.029647 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 09:03:28.029701 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":398,"threads_started":1}
I20260928 09:03:28.030534 21941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:28.031033 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:28.032029 21941 catalog_manager.cc:1347] Generated new cluster ID: f929d8ace31d439584c31f47caa0c6d4
I20260928 09:03:28.032076 21941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:28.033701 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:28.032589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:49609 (local address 127.0.0.1:44391)
0928 09:03:28.032626 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:28.032627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:28.032765 (+ 138us) server_negotiation.cc:407] Connection header received
0928 09:03:28.032818 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:28.032819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:28.032825 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:28.032902 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:28.033487 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:28.033488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:28.033488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:28.033492 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:28.033508 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:28.033513 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:28.033583 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:28.033626 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 09:03:28.033639 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:03:28.050562 21941 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:28.050756 21941 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:28.069437 21941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f2d4a296d841029029ac2f527464da: Generated new TSK 0
I20260928 09:03:28.069680 21941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:28.077702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21856
2026-09-28T09:03:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:28 dist-test-slave-286n krb5kdc[21955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:28 dist-test-slave-286n krb5kdc[21955](info): set up 2 sockets
Sep 28 09:03:28 dist-test-slave-286n krb5kdc[21955](info): commencing operation
krb5kdc: starting...
W20260928 09:03:29.128854 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:29 dist-test-slave-286n krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:29.247542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:29.328135 21968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:29.328359 21968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:29.328382 21968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:29.329578 21968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:29.329624 21968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:29.329638 21968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:29.329651 21968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:29.329663 21968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:29.331137 21968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:29.331384 21968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:29.331669 21968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:29.336545 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:29.336535 21974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:29.336717 21968 server_base.cc:1034] running on GCE node
W20260928 09:03:29.336782 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:29.337034 21968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:29.337352 21968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:29.338531 21968 hybrid_clock.cc:648] HybridClock initialized: now 1790586209338502 us; error 53 us; skew 500 ppm
Sep 28 09:03:29 dist-test-slave-286n krb5kdc[21955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:29.341289 21968 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:29.341776 21968 webserver.cc:460] Webserver started at http://127.0.98.62:46805/ using document root <none> and password file <none>
I20260928 09:03:29.342022 21968 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:29.342072 21968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:29.342250 21968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:29.343361 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "f4e84136a4224f2f869779f9e3874eb6"
format_stamp: "Formatted at 2026-09-28 09:03:29 on dist-test-slave-286n"
server_key: "d70d7a587f690d464b1d621ece7a9b81"
server_key_iv: "4218bdb3ba10eb1e079f63abd807b70b"
server_key_version: "encryptionkey@0"
I20260928 09:03:29.343739 21968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4e84136a4224f2f869779f9e3874eb6"
format_stamp: "Formatted at 2026-09-28 09:03:29 on dist-test-slave-286n"
server_key: "d70d7a587f690d464b1d621ece7a9b81"
server_key_iv: "4218bdb3ba10eb1e079f63abd807b70b"
server_key_version: "encryptionkey@0"
I20260928 09:03:29.345202 21968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:29.345969 21983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:29.346148 21968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:29.346244 21968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "f4e84136a4224f2f869779f9e3874eb6"
format_stamp: "Formatted at 2026-09-28 09:03:29 on dist-test-slave-286n"
server_key: "d70d7a587f690d464b1d621ece7a9b81"
server_key_iv: "4218bdb3ba10eb1e079f63abd807b70b"
server_key_version: "encryptionkey@0"
I20260928 09:03:29.346321 21968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:29.371384 21968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:29.374015 21968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:29.374188 21968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:29.378597 21968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41317
I20260928 09:03:29.378672 22035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41317 every 8 connection(s)
I20260928 09:03:29.379037 21968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:29.379930 22036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:29.382371 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6: Bootstrap starting.
I20260928 09:03:29.382431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 21968
I20260928 09:03:29.382522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:29.382731 392 external_mini_cluster.cc:1428] Setting key fd2750725543276c61374834e450b1ab
I20260928 09:03:29.383112 22036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:29.383514 22036 log.cc:824] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:29.384230 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6: No bootstrap required, opened a new log
I20260928 09:03:29.386547 22036 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e84136a4224f2f869779f9e3874eb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260928 09:03:29.386723 22036 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:29.386755 22036 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e84136a4224f2f869779f9e3874eb6, State: Initialized, Role: FOLLOWER
I20260928 09:03:29.386878 22036 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e84136a4224f2f869779f9e3874eb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260928 09:03:29.386971 22036 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:29.387008 22036 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:29.387050 22036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:29.387653 22036 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e84136a4224f2f869779f9e3874eb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260928 09:03:29.387776 22036 leader_election.cc:302] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e84136a4224f2f869779f9e3874eb6; no voters:
I20260928 09:03:29.387984 22036 leader_election.cc:288] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:29.388007 22041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:29.388240 22041 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [term 1 LEADER]: Becoming Leader. State: Replica: f4e84136a4224f2f869779f9e3874eb6, State: Running, Role: LEADER
I20260928 09:03:29.388289 22036 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:29.388367 22041 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e84136a4224f2f869779f9e3874eb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } }
I20260928 09:03:29.388756 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e84136a4224f2f869779f9e3874eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e84136a4224f2f869779f9e3874eb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } }
I20260928 09:03:29.388748 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e84136a4224f2f869779f9e3874eb6. Latest consensus state: current_term: 1 leader_uuid: "f4e84136a4224f2f869779f9e3874eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e84136a4224f2f869779f9e3874eb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } }
I20260928 09:03:29.388911 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:29.388919 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:29.389660 22050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 09:03:29 dist-test-slave-286n krb5kdc[21955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:29.390287 22050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:29.391840 22050 catalog_manager.cc:1347] Generated new cluster ID: 2e919a1ed3534ad9a9718d9d1b9b8953
I20260928 09:03:29.391894 22050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:29.394176 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:29.383280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45269 (local address 127.0.98.62:41317)
0928 09:03:29.383471 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:29.383479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:29.383504 (+ 25us) server_negotiation.cc:407] Connection header received
0928 09:03:29.383980 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:29.383987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:29.384044 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:29.384166 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:29.384988 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:29.385889 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:29.387029 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:29.387502 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:29.387592 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:29.390655 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:29.390660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:29.390660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:29.390678 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:29.392545 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:29.392974 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:29.392977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:29.392977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:29.393070 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:29.393294 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:29.393296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:29.393296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:29.393535 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:29.393715 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:29.393841 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 09:03:29.393955 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260928 09:03:29.395121 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:29.401239 22050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:29.401938 22050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:29.403083 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:29.396570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41847 (local address 127.0.98.62:41317)
0928 09:03:29.396618 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:29.396621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:29.396691 (+ 70us) server_negotiation.cc:407] Connection header received
0928 09:03:29.396795 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:29.396796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:29.396806 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:29.396850 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:29.397368 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:29.397871 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:29.398582 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:29.398815 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:29.398896 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:29.400160 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:29.400161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:29.400162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:29.400169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:29.401765 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:29.402274 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:29.402275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:29.402275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:29.402338 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:29.402562 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:29.402563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:29.402564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:29.402680 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:29.402790 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:29.402908 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 09:03:29.402972 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 09:03:29.411689 22050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4e84136a4224f2f869779f9e3874eb6: Generated new TSK 0
I20260928 09:03:29.411902 22050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:29.419942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 21968
W20260928 09:03:29.424337 21953 connection.cc:535] client connection to 127.0.98.62:41317 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41317): Connection reset by peer (error 104)
2026-09-28T09:03:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:29 dist-test-slave-286n krb5kdc[22067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:29 dist-test-slave-286n krb5kdc[22067](info): set up 2 sockets
Sep 28 09:03:29 dist-test-slave-286n krb5kdc[22067](info): commencing operation
krb5kdc: starting...
W20260928 09:03:30.456980 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:30 dist-test-slave-286n krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:30.575801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:30.656736 22080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:30.656994 22080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:30.657032 22080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:30.658187 22080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:30.658262 22080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:30.658288 22080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:30.658310 22080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:30.658341 22080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:30.659868 22080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:30.660154 22080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:30.660382 22080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:30.665702 22085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:30.665725 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:30.665697 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:30.666072 22080 server_base.cc:1034] running on GCE node
I20260928 09:03:30.666290 22080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:30.666534 22080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:30.667672 22080 hybrid_clock.cc:648] HybridClock initialized: now 1790586210667659 us; error 30 us; skew 500 ppm
Sep 28 09:03:30 dist-test-slave-286n krb5kdc[22067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:30.670264 22080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:30.670744 22080 webserver.cc:460] Webserver started at http://127.0.98.62:45461/ using document root <none> and password file <none>
I20260928 09:03:30.670985 22080 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:30.671051 22080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:30.671202 22080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:30.672261 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "c3771e37c3b748aba6bfcca3b530ef1b"
format_stamp: "Formatted at 2026-09-28 09:03:30 on dist-test-slave-286n"
server_key: "c67b8bbd094b59e0bd5b52c08ae82950"
server_key_iv: "651424cdbdeef98da6207db7e1b003fe"
server_key_version: "encryptionkey@0"
I20260928 09:03:30.672705 22080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3771e37c3b748aba6bfcca3b530ef1b"
format_stamp: "Formatted at 2026-09-28 09:03:30 on dist-test-slave-286n"
server_key: "c67b8bbd094b59e0bd5b52c08ae82950"
server_key_iv: "651424cdbdeef98da6207db7e1b003fe"
server_key_version: "encryptionkey@0"
I20260928 09:03:30.674075 22080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:30.674896 22095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:30.675081 22080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:30.675177 22080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "c3771e37c3b748aba6bfcca3b530ef1b"
format_stamp: "Formatted at 2026-09-28 09:03:30 on dist-test-slave-286n"
server_key: "c67b8bbd094b59e0bd5b52c08ae82950"
server_key_iv: "651424cdbdeef98da6207db7e1b003fe"
server_key_version: "encryptionkey@0"
I20260928 09:03:30.675263 22080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:30.690048 22080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:30.692150 22080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:30.692327 22080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:30.696588 22080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42499
I20260928 09:03:30.696642 22147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42499 every 8 connection(s)
I20260928 09:03:30.697023 22080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:30.697904 22148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:30.700193 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b: Bootstrap starting.
I20260928 09:03:30.700534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22080
I20260928 09:03:30.700876 22148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:30.700877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:30.701141 392 external_mini_cluster.cc:1428] Setting key ec51a197236173ca977178eaa0c2037a
I20260928 09:03:30.701205 22148 log.cc:824] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:30.702041 22148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b: No bootstrap required, opened a new log
I20260928 09:03:30.704720 22148 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260928 09:03:30.704876 22148 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:30.704900 22148 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3771e37c3b748aba6bfcca3b530ef1b, State: Initialized, Role: FOLLOWER
I20260928 09:03:30.705039 22148 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260928 09:03:30.705111 22148 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:30.705150 22148 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:30.705199 22148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:30.705931 22148 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260928 09:03:30.706043 22148 leader_election.cc:302] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3771e37c3b748aba6bfcca3b530ef1b; no voters:
I20260928 09:03:30.706231 22148 leader_election.cc:288] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:30.706305 22153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:30.706534 22148 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:30.706553 22153 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [term 1 LEADER]: Becoming Leader. State: Replica: c3771e37c3b748aba6bfcca3b530ef1b, State: Running, Role: LEADER
I20260928 09:03:30.707103 22153 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } }
I20260928 09:03:30.708050 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:30.701635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44603 (local address 127.0.98.62:42499)
0928 09:03:30.701795 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:30.701805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:30.701842 (+ 37us) server_negotiation.cc:407] Connection header received
0928 09:03:30.702468 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:30.702477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:30.702541 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:30.702697 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:30.702934 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:30.704380 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:30.705496 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:30.706157 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:30.706282 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:30.707195 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:30.707200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:30.707200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:30.707458 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:30.707496 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:30.707511 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:30.707611 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:30.707743 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 09:03:30.707773 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 09:03:30.708050 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } } }
I20260928 09:03:30.708436 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:30.708591 22154 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3771e37c3b748aba6bfcca3b530ef1b. Latest consensus state: current_term: 1 leader_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3771e37c3b748aba6bfcca3b530ef1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42499 } } }
I20260928 09:03:30.708743 22168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:30.708953 22154 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:30.709059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:30.709455 22168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:30.711110 22168 catalog_manager.cc:1347] Generated new cluster ID: ec266fa57e5a4663b1360a8e5ebd3547
I20260928 09:03:30.711158 22168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:30.713990 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:30.710479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60989 (local address 127.0.98.62:42499)
0928 09:03:30.710520 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:30.710523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:30.710553 (+ 30us) server_negotiation.cc:407] Connection header received
0928 09:03:30.710646 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:30.710647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:30.710658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:30.710688 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:30.710883 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:30.711609 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:30.712605 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:30.712959 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:30.713032 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:30.713747 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:30.713748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:30.713749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:30.713754 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:30.713772 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:30.713776 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:30.713841 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:30.713906 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 09:03:30.713917 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:03:30.739470 22168 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:30.740248 22168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:30.755506 22168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3771e37c3b748aba6bfcca3b530ef1b: Generated new TSK 0
I20260928 09:03:30.755738 22168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:30.762878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22080
W20260928 09:03:30.766772 22065 connection.cc:535] client connection to 127.0.98.62:42499 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42499): Connection reset by peer (error 104)
2026-09-28T09:03:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:30 dist-test-slave-286n krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:30 dist-test-slave-286n krb5kdc[22179](info): set up 2 sockets
Sep 28 09:03:30 dist-test-slave-286n krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260928 09:03:31.784930 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:31 dist-test-slave-286n krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:31.907738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:31.990356 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:31.990635 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:31.990682 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:31.991916 22192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:31.991997 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:31.992025 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:31.992058 22192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:31.992121 22192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:31.993695 22192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:31.994001 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:31.994251 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:31.999320 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:31.999320 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:31.999347 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:31.999468 22192 server_base.cc:1034] running on GCE node
I20260928 09:03:31.999811 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:32.000164 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:32.001362 22192 hybrid_clock.cc:648] HybridClock initialized: now 1790586212001341 us; error 48 us; skew 500 ppm
Sep 28 09:03:32 dist-test-slave-286n krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:32.017184 22192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:32.017704 22192 webserver.cc:460] Webserver started at http://127.0.98.62:44361/ using document root <none> and password file <none>
I20260928 09:03:32.017966 22192 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:32.018038 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:32.018199 22192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:32.019299 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "ed56138ff2ef482396a770b3e455fe47"
format_stamp: "Formatted at 2026-09-28 09:03:32 on dist-test-slave-286n"
server_key: "f96120acb646e14b48e23e2e1b43a484"
server_key_iv: "517ad0d392c0de48ec8d4a87a92799f4"
server_key_version: "encryptionkey@0"
I20260928 09:03:32.019677 22192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed56138ff2ef482396a770b3e455fe47"
format_stamp: "Formatted at 2026-09-28 09:03:32 on dist-test-slave-286n"
server_key: "f96120acb646e14b48e23e2e1b43a484"
server_key_iv: "517ad0d392c0de48ec8d4a87a92799f4"
server_key_version: "encryptionkey@0"
I20260928 09:03:32.021091 22192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 09:03:32.021919 22207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:32.022118 22192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:32.022225 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "ed56138ff2ef482396a770b3e455fe47"
format_stamp: "Formatted at 2026-09-28 09:03:32 on dist-test-slave-286n"
server_key: "f96120acb646e14b48e23e2e1b43a484"
server_key_iv: "517ad0d392c0de48ec8d4a87a92799f4"
server_key_version: "encryptionkey@0"
I20260928 09:03:32.022298 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:32.025704 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:32.027817 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:32.027976 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:32.032223 22192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42329
I20260928 09:03:32.032282 22259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42329 every 8 connection(s)
I20260928 09:03:32.032727 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:32.033707 22260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:32.035977 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47: Bootstrap starting.
I20260928 09:03:32.036562 22260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:32.036885 22260 log.cc:824] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:32.037547 22260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47: No bootstrap required, opened a new log
I20260928 09:03:32.039148 22260 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed56138ff2ef482396a770b3e455fe47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260928 09:03:32.039286 22260 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:32.039310 22260 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed56138ff2ef482396a770b3e455fe47, State: Initialized, Role: FOLLOWER
I20260928 09:03:32.039492 22260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed56138ff2ef482396a770b3e455fe47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260928 09:03:32.039572 22260 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:32.039614 22260 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:32.039662 22260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:32.040181 22260 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed56138ff2ef482396a770b3e455fe47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260928 09:03:32.040290 22260 leader_election.cc:302] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed56138ff2ef482396a770b3e455fe47; no voters:
I20260928 09:03:32.040472 22260 leader_election.cc:288] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:32.040561 22263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:32.040759 22260 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:32.040755 22263 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [term 1 LEADER]: Becoming Leader. State: Replica: ed56138ff2ef482396a770b3e455fe47, State: Running, Role: LEADER
I20260928 09:03:32.040977 22263 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed56138ff2ef482396a770b3e455fe47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260928 09:03:32.041662 22265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed56138ff2ef482396a770b3e455fe47. Latest consensus state: current_term: 1 leader_uuid: "ed56138ff2ef482396a770b3e455fe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed56138ff2ef482396a770b3e455fe47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } } }
I20260928 09:03:32.041749 22265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:32.041903 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed56138ff2ef482396a770b3e455fe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed56138ff2ef482396a770b3e455fe47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } } }
I20260928 09:03:32.042040 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:32.042150 22273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:32.042688 22273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:32.042848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22192
I20260928 09:03:32.042961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:32.043147 392 external_mini_cluster.cc:1428] Setting key d34b0a869c6ccb6162c8140431698eae
I20260928 09:03:32.044291 22273 catalog_manager.cc:1347] Generated new cluster ID: 3e8a73f04c774aa8bcee25ff26e902f1
I20260928 09:03:32.044343 22273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:32.045917 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:32.043617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33171 (local address 127.0.98.62:42329)
0928 09:03:32.043775 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:32.043785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:32.044061 (+ 276us) server_negotiation.cc:407] Connection header received
0928 09:03:32.044253 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:32.044261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:32.044305 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:32.044452 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:32.045470 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:32.045473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:32.045474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:32.045480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:32.045522 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:32.045537 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:32.045618 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:32.045700 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 09:03:32.045743 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260928 09:03:32.049801 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:32.049968 22273 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:32.050112 22273 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:32.052487 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:32.051431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37325 (local address 127.0.98.62:42329)
0928 09:03:32.051470 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:32.051471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:32.051531 (+ 60us) server_negotiation.cc:407] Connection header received
0928 09:03:32.051614 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:32.051615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:32.051619 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:32.051666 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:32.052277 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:32.052278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:32.052278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:32.052282 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:32.052298 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:32.052302 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:32.052355 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:32.052411 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 09:03:32.052425 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:03:32.072180 22273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed56138ff2ef482396a770b3e455fe47: Generated new TSK 0
I20260928 09:03:32.072482 22273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:32.082473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22192
2026-09-28T09:03:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:32 dist-test-slave-286n krb5kdc[22291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:32 dist-test-slave-286n krb5kdc[22291](info): set up 2 sockets
Sep 28 09:03:32 dist-test-slave-286n krb5kdc[22291](info): commencing operation
krb5kdc: starting...
W20260928 09:03:33.108992 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:33 dist-test-slave-286n krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:33.227680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:33.309047 22304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:33.309301 22304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:33.309340 22304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:33.310513 22304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:33.310586 22304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:33.310611 22304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:33.310640 22304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:33.310700 22304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:33.312178 22304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:33.312449 22304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:33.312703 22304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 09:03:33.318192 22304 server_base.cc:1034] running on GCE node
W20260928 09:03:33.318167 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:33.318149 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:33.318159 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:33.318673 22304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:33.318986 22304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:33.320179 22304 hybrid_clock.cc:648] HybridClock initialized: now 1790586213320139 us; error 50 us; skew 500 ppm
Sep 28 09:03:33 dist-test-slave-286n krb5kdc[22291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:33.323083 22304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:33.323534 22304 webserver.cc:460] Webserver started at http://127.0.98.62:45169/ using document root <none> and password file <none>
I20260928 09:03:33.323798 22304 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:33.323871 22304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:33.324023 22304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:33.325171 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "fdef9b86db4344c7b0397e9bce8e1ba6"
format_stamp: "Formatted at 2026-09-28 09:03:33 on dist-test-slave-286n"
server_key: "215a937a16560e8c08cf773eeacb46c8"
server_key_iv: "164d68c80a6ba13cbc930d928fcdbf6d"
server_key_version: "encryptionkey@0"
I20260928 09:03:33.325536 22304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdef9b86db4344c7b0397e9bce8e1ba6"
format_stamp: "Formatted at 2026-09-28 09:03:33 on dist-test-slave-286n"
server_key: "215a937a16560e8c08cf773eeacb46c8"
server_key_iv: "164d68c80a6ba13cbc930d928fcdbf6d"
server_key_version: "encryptionkey@0"
I20260928 09:03:33.326853 22304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:33.327554 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:33.327719 22304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:33.327831 22304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "fdef9b86db4344c7b0397e9bce8e1ba6"
format_stamp: "Formatted at 2026-09-28 09:03:33 on dist-test-slave-286n"
server_key: "215a937a16560e8c08cf773eeacb46c8"
server_key_iv: "164d68c80a6ba13cbc930d928fcdbf6d"
server_key_version: "encryptionkey@0"
I20260928 09:03:33.327903 22304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:33.336750 22304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:33.339056 22304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:33.339218 22304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:33.343209 22304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38495
I20260928 09:03:33.343238 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38495 every 8 connection(s)
I20260928 09:03:33.343621 22304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:33.344473 22372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:33.346700 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6: Bootstrap starting.
I20260928 09:03:33.347203 22372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:33.347429 22372 log.cc:824] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:33.348099 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6: No bootstrap required, opened a new log
I20260928 09:03:33.349669 22372 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } }
I20260928 09:03:33.349872 22372 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:33.349931 22372 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdef9b86db4344c7b0397e9bce8e1ba6, State: Initialized, Role: FOLLOWER
I20260928 09:03:33.350071 22372 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } }
I20260928 09:03:33.350157 22372 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:33.350196 22372 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:33.350247 22372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:33.350780 22372 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } }
I20260928 09:03:33.350893 22372 leader_election.cc:302] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdef9b86db4344c7b0397e9bce8e1ba6; no voters:
I20260928 09:03:33.351064 22372 leader_election.cc:288] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:33.351118 22375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:33.351289 22375 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [term 1 LEADER]: Becoming Leader. State: Replica: fdef9b86db4344c7b0397e9bce8e1ba6, State: Running, Role: LEADER
I20260928 09:03:33.351322 22372 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:33.351406 22375 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } }
I20260928 09:03:33.351787 22377 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdef9b86db4344c7b0397e9bce8e1ba6. Latest consensus state: current_term: 1 leader_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } }
I20260928 09:03:33.351929 22377 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:33.351929 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdef9b86db4344c7b0397e9bce8e1ba6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38495 } } }
I20260928 09:03:33.352140 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:33.352463 22384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:33.352593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22304
I20260928 09:03:33.352733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:33.352960 392 external_mini_cluster.cc:1428] Setting key 0b70b9503c7c24a622e55d14c0e16ce2
I20260928 09:03:33.353122 22384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:33.354800 22384 catalog_manager.cc:1347] Generated new cluster ID: 8cd60bb15d55405d949a45e387198cc1
I20260928 09:03:33.354861 22384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 09:03:33 dist-test-slave-286n krb5kdc[22291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:33.364413 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:33.353600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39775 (local address 127.0.98.62:38495)
0928 09:03:33.353770 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:33.353779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:33.353812 (+ 33us) server_negotiation.cc:407] Connection header received
0928 09:03:33.354039 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:33.354046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:33.354099 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:33.354242 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:33.354997 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:33.356396 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:33.357498 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:33.358109 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:33.358176 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:33.360934 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:33.360938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:33.360939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:33.360951 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:33.362808 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:33.363147 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:33.363150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:33.363150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:33.363242 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:33.363427 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:33.363428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:33.363428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:33.363712 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:33.363817 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:33.364036 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 09:03:33.364180 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260928 09:03:33.365790 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:33.366469 22384 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:33.367148 22384 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:33.374523 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:33.367441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51103 (local address 127.0.98.62:38495)
0928 09:03:33.367475 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:33.367477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:33.367555 (+ 78us) server_negotiation.cc:407] Connection header received
0928 09:03:33.367638 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:33.367639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:33.367648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:33.367682 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:33.368335 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:33.369116 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:33.370132 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:33.370452 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:33.370486 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:33.371855 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:33.371856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:33.371857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:33.371863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:33.373408 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:33.373775 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:33.373776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:33.373776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:33.373826 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:33.373985 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:33.373986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:33.373986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:33.374105 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:33.374180 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:33.374357 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 09:03:33.374433 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:03:33.388358 22384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdef9b86db4344c7b0397e9bce8e1ba6: Generated new TSK 0
I20260928 09:03:33.388592 22384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:33.393019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22304
2026-09-28T09:03:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:33 dist-test-slave-286n krb5kdc[22403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:33 dist-test-slave-286n krb5kdc[22403](info): set up 2 sockets
Sep 28 09:03:33 dist-test-slave-286n krb5kdc[22403](info): commencing operation
krb5kdc: starting...
W20260928 09:03:34.496855 392 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:34 dist-test-slave-286n krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:34.613739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33615
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:34.693370 22416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:34.693576 22416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:34.693600 22416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:34.694773 22416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:34.694821 22416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:34.694835 22416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:34.694849 22416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:34.694860 22416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:34.696343 22416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:34.696601 22416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:34.696868 22416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:34.701786 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:34.701792 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:34.701820 22416 server_base.cc:1034] running on GCE node
W20260928 09:03:34.701802 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:34.702297 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:34.702555 22416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:34.703697 22416 hybrid_clock.cc:648] HybridClock initialized: now 1790586214703675 us; error 41 us; skew 500 ppm
Sep 28 09:03:34 dist-test-slave-286n krb5kdc[22403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:34.706357 22416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:34.706853 22416 webserver.cc:460] Webserver started at http://127.0.98.62:42571/ using document root <none> and password file <none>
I20260928 09:03:34.707106 22416 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:34.707176 22416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:34.707332 22416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:34.708402 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "d3a46049acc9444c8378e561ab694960"
format_stamp: "Formatted at 2026-09-28 09:03:34 on dist-test-slave-286n"
server_key: "545b3baecc36284f11d831dd3dd2a412"
server_key_iv: "df73df5f2eb0091b64152ae38e1202c8"
server_key_version: "encryptionkey@0"
I20260928 09:03:34.708798 22416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3a46049acc9444c8378e561ab694960"
format_stamp: "Formatted at 2026-09-28 09:03:34 on dist-test-slave-286n"
server_key: "545b3baecc36284f11d831dd3dd2a412"
server_key_iv: "df73df5f2eb0091b64152ae38e1202c8"
server_key_version: "encryptionkey@0"
I20260928 09:03:34.710192 22416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:34.710950 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:34.711150 22416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:34.711256 22416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "d3a46049acc9444c8378e561ab694960"
format_stamp: "Formatted at 2026-09-28 09:03:34 on dist-test-slave-286n"
server_key: "545b3baecc36284f11d831dd3dd2a412"
server_key_iv: "df73df5f2eb0091b64152ae38e1202c8"
server_key_version: "encryptionkey@0"
I20260928 09:03:34.711344 22416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:34.724054 22416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:34.726207 22416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:34.726369 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:34.730885 22416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33615
I20260928 09:03:34.730926 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33615 every 8 connection(s)
I20260928 09:03:34.731314 22416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:34.732259 22484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:34.734673 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960: Bootstrap starting.
I20260928 09:03:34.735255 22484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:34.735548 22484 log.cc:824] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:34.736253 22484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960: No bootstrap required, opened a new log
I20260928 09:03:34.738013 22484 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a46049acc9444c8378e561ab694960" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260928 09:03:34.738157 22484 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:34.738181 22484 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a46049acc9444c8378e561ab694960, State: Initialized, Role: FOLLOWER
I20260928 09:03:34.738392 22484 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a46049acc9444c8378e561ab694960" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260928 09:03:34.738457 22484 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:34.738480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22416
I20260928 09:03:34.738516 22484 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:34.738561 22484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:34.738596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:34.738806 392 external_mini_cluster.cc:1428] Setting key 7e711184e61c02653bf21bf717f88e38
I20260928 09:03:34.739229 22484 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a46049acc9444c8378e561ab694960" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260928 09:03:34.739410 22484 leader_election.cc:302] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a46049acc9444c8378e561ab694960; no voters:
I20260928 09:03:34.739657 22484 leader_election.cc:288] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:34.739684 22489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:34.739795 22489 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [term 1 LEADER]: Becoming Leader. State: Replica: d3a46049acc9444c8378e561ab694960, State: Running, Role: LEADER
I20260928 09:03:34.739931 22484 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:34.739935 22489 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a46049acc9444c8378e561ab694960" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260928 09:03:34.740404 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a46049acc9444c8378e561ab694960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a46049acc9444c8378e561ab694960" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } } }
I20260928 09:03:34.740417 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a46049acc9444c8378e561ab694960. Latest consensus state: current_term: 1 leader_uuid: "d3a46049acc9444c8378e561ab694960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a46049acc9444c8378e561ab694960" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } } }
I20260928 09:03:34.740504 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:34.740535 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960 [sys.catalog]: This master's current role is: LEADER
W20260928 09:03:34.741811 22505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:34.741884 22505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:34.741966 22496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:34.742141 22496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:34.743619 22496 catalog_manager.cc:1347] Generated new cluster ID: a0596ea48c244c32a6213f7c41db64d1
I20260928 09:03:34.743673 22496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:34.745158 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:34.739317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35769 (local address 127.0.98.62:33615)
0928 09:03:34.739526 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:34.739536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:34.739574 (+ 38us) server_negotiation.cc:407] Connection header received
0928 09:03:34.739671 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:34.739677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:34.739743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:34.740116 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:34.740545 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:34.742001 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:34.743064 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:34.743495 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:34.743985 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:34.744441 (+ 456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:34.744444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:34.744445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:34.744684 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:34.744724 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:34.744735 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:34.744826 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:34.744902 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 09:03:34.744965 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260928 09:03:34.749922 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:34.756366 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:34.751609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35215 (local address 127.0.98.62:33615)
0928 09:03:34.751656 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:34.751659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:34.752858 (+ 1199us) server_negotiation.cc:407] Connection header received
0928 09:03:34.753046 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:34.753047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:34.753063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:34.753128 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:34.753299 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:34.753877 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:34.754796 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:34.755049 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:34.755152 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:34.756023 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:34.756024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:34.756025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:34.756032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:34.756048 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:34.756054 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:34.756139 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:34.756214 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 09:03:34.756243 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:34.760039 22496 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:34.760872 22496 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:34.772158 22496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a46049acc9444c8378e561ab694960: Generated new TSK 0
I20260928 09:03:34.772424 22496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:34.778165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22416
2026-09-28T09:03:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:34 dist-test-slave-286n krb5kdc[22515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:34 dist-test-slave-286n krb5kdc[22515](info): set up 2 sockets
Sep 28 09:03:34 dist-test-slave-286n krb5kdc[22515](info): commencing operation
krb5kdc: starting...
W20260928 09:03:35.800859 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:35 dist-test-slave-286n krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:35.918958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:35.998301 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:35.998576 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:35.998620 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:35.999845 22528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:35.999923 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:35.999948 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:35.999977 22528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:36.000010 22528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:36.001539 22528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:36.001839 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:36.002074 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:36.006995 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:36.007155 22528 server_base.cc:1034] running on GCE node
W20260928 09:03:36.007009 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:36.006980 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:36.007515 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:36.007807 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:36.008980 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790586216008969 us; error 45 us; skew 500 ppm
Sep 28 09:03:36 dist-test-slave-286n krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:36.011344 22528 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:36.011801 22528 webserver.cc:460] Webserver started at http://127.0.98.62:35717/ using document root <none> and password file <none>
I20260928 09:03:36.012023 22528 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:36.012089 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:36.012238 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:36.013350 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "23840544a6e64a73ae8d237bb1a6dcbb"
format_stamp: "Formatted at 2026-09-28 09:03:36 on dist-test-slave-286n"
server_key: "8d85738b723b5268733ca7bf7a22a8b1"
server_key_iv: "3603bb6b816c6b0cc48e4a9a70e9fa0c"
server_key_version: "encryptionkey@0"
I20260928 09:03:36.013705 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "23840544a6e64a73ae8d237bb1a6dcbb"
format_stamp: "Formatted at 2026-09-28 09:03:36 on dist-test-slave-286n"
server_key: "8d85738b723b5268733ca7bf7a22a8b1"
server_key_iv: "3603bb6b816c6b0cc48e4a9a70e9fa0c"
server_key_version: "encryptionkey@0"
I20260928 09:03:36.015030 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 09:03:36.015743 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:36.015913 22528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 09:03:36.016010 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "23840544a6e64a73ae8d237bb1a6dcbb"
format_stamp: "Formatted at 2026-09-28 09:03:36 on dist-test-slave-286n"
server_key: "8d85738b723b5268733ca7bf7a22a8b1"
server_key_iv: "3603bb6b816c6b0cc48e4a9a70e9fa0c"
server_key_version: "encryptionkey@0"
I20260928 09:03:36.016093 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:36.019371 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:36.021453 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:36.021610 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:36.025651 22528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39301
I20260928 09:03:36.025681 22595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39301 every 8 connection(s)
I20260928 09:03:36.026005 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:36.026973 22596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:36.029202 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb: Bootstrap starting.
I20260928 09:03:36.029712 22596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:36.029942 22596 log.cc:824] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:36.030555 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb: No bootstrap required, opened a new log
I20260928 09:03:36.032058 22596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 09:03:36.032186 22596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 09:03:36.032208 22596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23840544a6e64a73ae8d237bb1a6dcbb, State: Initialized, Role: FOLLOWER
I20260928 09:03:36.032348 22596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 09:03:36.032428 22596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 09:03:36.032451 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 09:03:36.032486 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:36.033010 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 09:03:36.033093 22596 leader_election.cc:302] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23840544a6e64a73ae8d237bb1a6dcbb; no voters:
I20260928 09:03:36.033203 22596 leader_election.cc:288] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 09:03:36.033255 22599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:36.033453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22528
I20260928 09:03:36.033457 22599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [term 1 LEADER]: Becoming Leader. State: Replica: 23840544a6e64a73ae8d237bb1a6dcbb, State: Running, Role: LEADER
I20260928 09:03:36.033463 22596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:36.033563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:36.033572 22599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 09:03:36.033748 392 external_mini_cluster.cc:1428] Setting key a7af59a15811784259168d955008829b
I20260928 09:03:36.034973 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23840544a6e64a73ae8d237bb1a6dcbb. Latest consensus state: current_term: 1 leader_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } } }
I20260928 09:03:36.035056 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:36.035300 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23840544a6e64a73ae8d237bb1a6dcbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } } }
I20260928 09:03:36.035449 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb [sys.catalog]: This master's current role is: LEADER
W20260928 09:03:36.035982 22617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:36.036053 22617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:36.036145 22614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:36.036346 22614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:36.036470 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:36.034220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35871 (local address 127.0.98.62:39301)
0928 09:03:36.034414 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:36.034426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:36.034672 (+ 246us) server_negotiation.cc:407] Connection header received
0928 09:03:36.034977 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:36.034985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:36.035018 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:36.035171 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:36.035978 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:36.035982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:36.035982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:36.035987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:36.036028 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:36.036044 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:36.036111 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:36.036218 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 09:03:36.036260 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
I20260928 09:03:36.037469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:36.037926 22614 catalog_manager.cc:1347] Generated new cluster ID: 95cb620184ef4c66b7fd3462157cafe8
I20260928 09:03:36.037978 22614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:36.039937 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:36.039015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38617 (local address 127.0.98.62:39301)
0928 09:03:36.039051 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:36.039052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:36.039061 (+ 9us) server_negotiation.cc:407] Connection header received
0928 09:03:36.039125 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:36.039126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:36.039131 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:36.039184 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:36.039733 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:36.039734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:36.039734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 09:03:36.039738 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:36.039751 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 09:03:36.039755 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:36.039804 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:36.039860 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 09:03:36.039874 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 09:03:36.054502 22614 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:36.054679 22614 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:36.062336 22614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23840544a6e64a73ae8d237bb1a6dcbb: Generated new TSK 0
I20260928 09:03:36.062569 22614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 09:03:36.065879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22528
2026-09-28T09:03:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1288 ms)
[----------] 18 tests from AuthTokenIssuingTest (23932 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:36 dist-test-slave-286n krb5kdc[22627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:36 dist-test-slave-286n krb5kdc[22627](info): set up 2 sockets
Sep 28 09:03:36 dist-test-slave-286n krb5kdc[22627](info): commencing operation
krb5kdc: starting...
W20260928 09:03:37.092886 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:37.211912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37191
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37191,127.0.98.61:40283,127.0.98.60:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:37.296186 22640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:03:37.296434 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:37.296465 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:37.296523 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:37.297709 22640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:37.297776 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:37.297801 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:37.297829 22640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:37.297861 22640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:37.299392 22640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37191,127.0.98.61:40283,127.0.98.60:32943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:37.299676 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:37.299918 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:37.304759 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:37.304785 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:37.305123 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:37.305356 22640 server_base.cc:1034] running on GCE node
I20260928 09:03:37.305581 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:37.305824 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:37.306962 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790586217306948 us; error 27 us; skew 500 ppm
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:37.313802 22640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:37.314298 22640 webserver.cc:460] Webserver started at http://127.0.98.62:33589/ using document root <none> and password file <none>
I20260928 09:03:37.314550 22640 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:37.314618 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:37.314762 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:37.315842 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "060970a08cde40109618980383585edd"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "6229c0db5a30c98e5bb85d90bdb5d5d9"
server_key_iv: "fdbb7ef6f020c0b2fef51d212cbf6904"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.316206 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "060970a08cde40109618980383585edd"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "6229c0db5a30c98e5bb85d90bdb5d5d9"
server_key_iv: "fdbb7ef6f020c0b2fef51d212cbf6904"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.317595 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:37.318297 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:37.318480 22640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:03:37.318579 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "060970a08cde40109618980383585edd"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "6229c0db5a30c98e5bb85d90bdb5d5d9"
server_key_iv: "fdbb7ef6f020c0b2fef51d212cbf6904"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.318651 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:37.326248 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:37.328016 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:37.328167 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:37.332149 22640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37191
I20260928 09:03:37.332183 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37191 every 8 connection(s)
I20260928 09:03:37.332563 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:37.333552 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:37.335052 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:37.336560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22640
I20260928 09:03:37.336689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:37.336877 392 external_mini_cluster.cc:1428] Setting key 4803eaf1701ae3a4719277ba979ffff3
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 09:03:37.345296 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.335701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.0.98.62:37191)
0928 09:03:37.335872 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.335902 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.336015 (+ 113us) server_negotiation.cc:407] Connection header received
0928 09:03:37.336600 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.336602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.336621 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.336748 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.337357 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.338159 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.339035 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.339350 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.339475 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.342034 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.342041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.342041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.342048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.343603 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.344048 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.344051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.344051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.344129 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.344404 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.344404 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.344404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.344618 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.344741 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.344984 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.345064 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260928 09:03:37.345283 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.335672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37191 (local address 127.0.0.1:55902)
0928 09:03:37.335862 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.335904 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.336442 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.336752 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.336755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.337098 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.337252 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.337257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.338257 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.338258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.338933 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.338935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.339466 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.339467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.339531 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.340113 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.340121 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.341945 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.343694 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.343702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.343705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.343966 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.344202 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.344203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.344203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.344332 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.344725 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.344731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.344792 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.344876 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.345009 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260928 09:03:37.346285 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 09:03:37.346688 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.346554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48192)
0928 09:03:37.346603 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.346655 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:03:37.346807 22659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
I20260928 09:03:37.347298 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.347179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48204)
0928 09:03:37.347230 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.347266 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:03:37.347442 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40283: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 09:03:37.361099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40283
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37191,127.0.98.61:40283,127.0.98.60:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:03:37.396744 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } attempt: 1
I20260928 09:03:37.397333 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.397180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48216)
0928 09:03:37.397252 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.397300 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 09:03:37.397774 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.397688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48224)
0928 09:03:37.397731 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.397757 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:03:37.397883 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40283: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
W20260928 09:03:37.441344 22715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:03:37.441610 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:37.441641 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:37.441723 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:37.442946 22715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:37.443018 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:37.443040 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:37.443063 22715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:37.443094 22715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:37.444586 22715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37191,127.0.98.61:40283,127.0.98.60:32943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:37.444912 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:37.445158 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:37.450023 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:37.450030 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:37.450134 22715 server_base.cc:1034] running on GCE node
W20260928 09:03:37.450030 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:37.450513 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:37.450785 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:37.451953 22715 hybrid_clock.cc:648] HybridClock initialized: now 1790586217451948 us; error 36 us; skew 500 ppm
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:37.454380 22715 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 09:03:37.454782 22715 webserver.cc:460] Webserver started at http://127.0.98.61:36539/ using document root <none> and password file <none>
I20260928 09:03:37.455022 22715 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:37.455087 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:37.455220 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:37.456250 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data/instance:
uuid: "4387312e3aeb4a9baf17860ecca645de"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "8f30ae95df61d4e5ef592a37e51d7352"
server_key_iv: "c7a0e525578839dc0db6f541776ffaa3"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.456590 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal/instance:
uuid: "4387312e3aeb4a9baf17860ecca645de"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "8f30ae95df61d4e5ef592a37e51d7352"
server_key_iv: "c7a0e525578839dc0db6f541776ffaa3"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.457970 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:37.458725 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:37.458923 22715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:37.459022 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal
uuid: "4387312e3aeb4a9baf17860ecca645de"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "8f30ae95df61d4e5ef592a37e51d7352"
server_key_iv: "c7a0e525578839dc0db6f541776ffaa3"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.459074 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:37.466029 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } attempt: 2
I20260928 09:03:37.466429 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.466291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48240)
0928 09:03:37.466354 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.466395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:03:37.466786 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.466697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48252)
0928 09:03:37.466738 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.466765 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 09:03:37.466910 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40283: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40283: connect: Connection refused (error 111)
I20260928 09:03:37.467231 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:37.469046 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:37.469193 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:37.473084 22715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40283
I20260928 09:03:37.473114 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40283 every 8 connection(s)
I20260928 09:03:37.473452 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:03:37.474375 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:37.475872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22715
I20260928 09:03:37.475980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-1/wal/instance
I20260928 09:03:37.475867 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:37.476167 392 external_mini_cluster.cc:1428] Setting key a51a84bff54bfecfc573001dcf375978
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 09:03:37.485985 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.476676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.0.98.62:37191)
0928 09:03:37.476717 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.476719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.476940 (+ 221us) server_negotiation.cc:407] Connection header received
0928 09:03:37.477481 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.477482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.477497 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.477570 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.478397 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.478881 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.480395 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.480586 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.480697 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.483113 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.483114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.483115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.483123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.484565 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.485007 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.485009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.485009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.485070 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.485320 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.485320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.485321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.485438 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.485508 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.485796 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.485890 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:03:37.486176 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.476663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37191 (local address 127.0.0.1:55918)
0928 09:03:37.476819 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.476855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.477342 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.477641 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.477645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.478006 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.478290 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.478295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.478971 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.478972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.480308 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.480311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.480703 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.480705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.480887 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.481441 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.481452 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.483009 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.484671 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.484679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.484681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.484927 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.485138 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.485139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.485139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.485247 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.485511 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.485519 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.485659 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.485763 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.485920 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 09:03:37.486793 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:37.495177 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.487087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48264)
0928 09:03:37.487145 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.487158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.487214 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.487543 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.487544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.487713 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.487763 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.487764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.488798 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.488799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.489309 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.489311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.489823 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.489824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.489888 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.490371 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.490377 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.491872 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.493899 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.493900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.493900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.494077 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.494321 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.494321 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.494322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.494381 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.494739 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.494740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.494780 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.494860 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.495011 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 09:03:37.495194 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.487245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.0.98.61:40283)
0928 09:03:37.487404 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.487408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.487424 (+ 16us) server_negotiation.cc:407] Connection header received
0928 09:03:37.487442 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.487443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.487454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.487544 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.487847 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.488696 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.489413 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.489729 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.489826 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.491983 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.491991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.491992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.491999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.493786 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.494185 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.494188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.494189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.494250 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.494500 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.494501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.494501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.494661 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.494763 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.494888 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.495096 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260928 09:03:37.495959 22783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:37.496323 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.496196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52830)
0928 09:03:37.496234 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.496281 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 09:03:37.496454 22734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
I20260928 09:03:37.497002 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.496851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52834)
0928 09:03:37.496922 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.496949 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 09:03:37.497107 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 09:03:37.501865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32943
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37191,127.0.98.61:40283,127.0.98.60:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:03:37.546466 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } attempt: 1
I20260928 09:03:37.546965 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.546822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52838)
0928 09:03:37.546888 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.546936 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:03:37.547302 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.547208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52844)
0928 09:03:37.547252 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.547278 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 09:03:37.547392 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
I20260928 09:03:37.558084 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } attempt: 3
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:37.565421 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.558522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.98.61:40283)
0928 09:03:37.558591 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.558593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.558604 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:03:37.558693 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.558694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.558703 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.558776 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.559138 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.559635 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.560350 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.560563 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.560674 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.562885 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.562887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.562887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.562894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.564220 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.564609 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.564610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.564611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.564711 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.564945 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.564946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.564946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.565053 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.565104 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.565281 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.565354 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:03:37.565526 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.558456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48278)
0928 09:03:37.558509 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.558527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.558606 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.558780 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.558781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.558988 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.559068 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.559071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.559724 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.559725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.560262 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.560263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.560672 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.560673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.560759 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.561232 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.561258 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.562796 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.564305 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.564306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.564306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.564537 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.564771 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.564772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.564773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.564872 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.565118 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.565119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.565179 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.565267 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.565381 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 09:03:37.565989 22708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:37.566370 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.566235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52850)
0928 09:03:37.566297 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.566338 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:03:37.566720 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.566607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52854)
0928 09:03:37.566662 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.566696 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 09:03:37.566843 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
W20260928 09:03:37.585096 22790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:03:37.585336 22790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:37.585356 22790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:37.585420 22790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:37.586598 22790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:37.586649 22790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:37.586665 22790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:37.586723 22790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:37.586747 22790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:37.588297 22790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37191,127.0.98.61:40283,127.0.98.60:32943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:37.588573 22790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:37.588865 22790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:37.593778 22795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:37.593926 22790 server_base.cc:1034] running on GCE node
W20260928 09:03:37.593839 22796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:37.593778 22798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:37.594311 22790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:37.594568 22790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:37.595714 22790 hybrid_clock.cc:648] HybridClock initialized: now 1790586217595696 us; error 29 us; skew 500 ppm
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:37.598004 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } attempt: 1
I20260928 09:03:37.598191 22790 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 09:03:37.598403 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.598255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52858)
0928 09:03:37.598335 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.598368 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 09:03:37.598794 22790 webserver.cc:460] Webserver started at http://127.0.98.60:44345/ using document root <none> and password file <none>
I20260928 09:03:37.598798 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.598669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52872)
0928 09:03:37.598720 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.598744 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 09:03:37.598937 22708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
I20260928 09:03:37.599100 22790 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:37.599169 22790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:37.599308 22790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:37.600414 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data/instance:
uuid: "9413fd1ae09848df992ffcf4e78d3cc4"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "eecada1c7123752444f9de015c999a89"
server_key_iv: "ef846072a2fa802261bcbf74092612b9"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.600840 22790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal/instance:
uuid: "9413fd1ae09848df992ffcf4e78d3cc4"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "eecada1c7123752444f9de015c999a89"
server_key_iv: "ef846072a2fa802261bcbf74092612b9"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.602293 22790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:37.603061 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:37.603281 22790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 09:03:37.603394 22790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal
uuid: "9413fd1ae09848df992ffcf4e78d3cc4"
format_stamp: "Formatted at 2026-09-28 09:03:37 on dist-test-slave-286n"
server_key: "eecada1c7123752444f9de015c999a89"
server_key_iv: "ef846072a2fa802261bcbf74092612b9"
server_key_version: "encryptionkey@0"
I20260928 09:03:37.603469 22790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:37.613744 22790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:37.615535 22790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:37.615531 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } attempt: 2
I20260928 09:03:37.615718 22790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:37.615954 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.615819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52874)
0928 09:03:37.615893 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.615928 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 09:03:37.616304 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.616193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52888)
0928 09:03:37.616238 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.616267 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 09:03:37.616392 22783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32943: connect: Connection refused (error 111)
I20260928 09:03:37.619695 22790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32943
I20260928 09:03:37.619740 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32943 every 8 connection(s)
I20260928 09:03:37.620098 22790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:03:37.621174 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:37.622717 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:37.626716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 22790
I20260928 09:03:37.626827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585920139418-392-0/minicluster-data/master-2/wal/instance
I20260928 09:03:37.627055 392 external_mini_cluster.cc:1428] Setting key c4e0f0365b095f0e6ed3f42b76b3b0a3
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:37.632064 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.623368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.98.62:37191)
0928 09:03:37.623433 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.623435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.623642 (+ 207us) server_negotiation.cc:407] Connection header received
0928 09:03:37.624155 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.624156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.624169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.624220 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.624896 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.625366 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.626328 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.626516 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.626597 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.629197 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.629199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.629199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.629207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.630704 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.631109 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.631111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.631111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.631175 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.631426 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.631427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.631427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.631545 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.631622 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.631892 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.631974 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 09:03:37.632270 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.623366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37191 (local address 127.0.0.1:55922)
0928 09:03:37.623526 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.623561 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.624041 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.624287 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.624292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.624535 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.624793 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.624798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.625455 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.625457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.626233 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.626235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.626618 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.626619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.626745 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.627426 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.627435 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.629113 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.630792 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.630798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.630799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.631009 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.631242 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.631243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.631243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.631355 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.631612 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.631619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.631756 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.631869 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.632009 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 09:03:37.632833 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:37.636283 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.627503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.0.98.60:32943)
0928 09:03:37.627665 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.627670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.627699 (+ 29us) server_negotiation.cc:407] Connection header received
0928 09:03:37.627769 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.627779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.627799 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.627890 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.628688 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.629545 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.630475 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.630952 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.631035 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.633485 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.633492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.633493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.633500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.635046 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.635467 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.635470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.635470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.635531 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.635725 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.635726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.635726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.635884 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.636041 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.636088 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.636168 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:37.639663 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.633190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.98.61:40283)
0928 09:03:37.633251 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.633253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.633261 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:37.633296 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.633297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.633306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.633355 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.633604 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.634060 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.634761 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.634989 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.635074 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.637139 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.637140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.637141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.637149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.638561 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.638916 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.638916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.638917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.638966 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.639143 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.639144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.639144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.639239 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.639318 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.639492 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.639561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 09:03:37.639899 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.633115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40283 (local address 127.0.0.1:48290)
0928 09:03:37.633188 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.633197 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.633237 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.633355 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.633356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.633489 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.633531 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.633532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.634149 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.634150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.634673 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.634675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.635073 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.635074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.635142 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.635530 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.635534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.637055 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.638637 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.638637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.638638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.638847 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.639026 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.639026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.639027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.639086 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.639324 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.639325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.639367 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.639463 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.639557 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 09:03:37.640301 22858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:37.645047 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.637254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.0.98.62:37191)
0928 09:03:37.637294 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.637296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.637303 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:03:37.637322 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.637324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.637332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.637400 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.637836 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.638286 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.639205 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.639410 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.639491 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.642049 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.642050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.642051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.642057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.643438 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.644127 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.644128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.644129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.644192 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.644362 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.644362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.644363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.644491 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.644693 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.644870 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.644947 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:37.647607 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.640561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52902)
0928 09:03:37.640613 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.640626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.640704 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.640814 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.640814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.640971 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.641006 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.641008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.641865 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.641866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.642609 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.642612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.643104 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.643105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.643152 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.643509 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.643513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.644831 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.646691 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.646692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.646693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.646907 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.647087 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.647088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.647089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.647150 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.647358 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.647359 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.647397 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.647468 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.647547 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:03:37.647740 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.640591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.0.98.60:32943)
0928 09:03:37.640621 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.640623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.640688 (+ 65us) server_negotiation.cc:407] Connection header received
0928 09:03:37.640760 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.640760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.640773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.640809 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.641090 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.641787 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.642687 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.643025 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.643101 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.644916 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.644917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.644917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.644924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.646606 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.646978 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.646978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.646979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.647026 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.647208 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.647208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.647209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.647301 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.647350 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.647502 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.647601 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":14}
I20260928 09:03:37.649657 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: Bootstrap starting.
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:37.650664 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:37.650970 22858 log.cc:824] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:37.651746 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: No bootstrap required, opened a new log
I20260928 09:03:37.653293 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.653442 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9413fd1ae09848df992ffcf4e78d3cc4, State: Initialized, Role: FOLLOWER
I20260928 09:03:37.653530 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.653695 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.645672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.98.61:40283)
0928 09:03:37.645742 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.645744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.645974 (+ 230us) server_negotiation.cc:407] Connection header received
0928 09:03:37.646039 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.646039 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.646047 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.646090 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.646692 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.647397 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.648470 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.648706 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.648776 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.651159 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.651160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.651160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.651167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.652556 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.652945 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.652947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.652947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.653002 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.653194 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.653195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.653195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.653303 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.653473 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.653508 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.653581 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 09:03:37.653896 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:37.653868 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.653991 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:37.655155 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 09:03:37.655726 22875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:37.655789 22875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:37.656698 22682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "060970a08cde40109618980383585edd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55936
I20260928 09:03:37.655997 22790 master_runner.cc:186] Error getting master registration for 127.0.98.62:37191: OK, instance_id { permanent_uuid: "060970a08cde40109618980383585edd" instance_seqno: 1790586217330988 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:03:37.658358 22790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:03:37.674094 22708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } attempt: 2
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:37.681936 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.674555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52904)
0928 09:03:37.674641 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.674670 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.674724 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.674875 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.674876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.675054 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.675099 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.675101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.675783 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.675784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.676289 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.676291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.676685 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.676686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.676742 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.677253 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.677259 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.678856 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.680735 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.680736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.680737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.680974 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.681182 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.681183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.681184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.681251 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.681593 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.681594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.681643 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.681717 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.681826 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 09:03:37.681942 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.674642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.0.98.60:32943)
0928 09:03:37.674711 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.674714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.674726 (+ 12us) server_negotiation.cc:407] Connection header received
0928 09:03:37.674787 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.674788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.674797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.674867 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.675193 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.675673 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.676386 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.676577 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.676686 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.678959 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.678960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.678961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.678968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.680619 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.681058 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.681060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.681060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.681118 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.681317 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.681318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.681318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.681516 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.681582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.681753 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.681842 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 09:03:37.683458 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd: Bootstrap starting.
I20260928 09:03:37.684121 22708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:37.684464 22708 log.cc:824] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:37.685242 22708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd: No bootstrap required, opened a new log
I20260928 09:03:37.686285 22708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.686401 22708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060970a08cde40109618980383585edd, State: Initialized, Role: FOLLOWER
I20260928 09:03:37.686486 22708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.686785 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.686839 22708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:37.686863 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:37.688138 22682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 09:03:37.688458 22682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 0 FOLLOWER]: Advancing to term 1
W20260928 09:03:37.688547 22889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:37.688611 22889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:37.689826 22682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.690404 22682 leader_election.cc:288] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [CANDIDATE]: Term 1 election: Requested vote from peers 4387312e3aeb4a9baf17860ecca645de (127.0.98.61:40283), 9413fd1ae09848df992ffcf4e78d3cc4 (127.0.98.60:32943)
I20260928 09:03:37.690847 22832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "060970a08cde40109618980383585edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9413fd1ae09848df992ffcf4e78d3cc4"
I20260928 09:03:37.690971 22832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:37.691900 22832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060970a08cde40109618980383585edd in term 1.
I20260928 09:03:37.692126 22659 leader_election.cc:302] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060970a08cde40109618980383585edd, 9413fd1ae09848df992ffcf4e78d3cc4; no voters:
I20260928 09:03:37.692235 22878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:37.692354 22878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [term 1 LEADER]: Becoming Leader. State: Replica: 060970a08cde40109618980383585edd, State: Running, Role: LEADER
I20260928 09:03:37.692433 22878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.692847 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 060970a08cde40109618980383585edd. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.692936 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:37.691884 22640 master_runner.cc:186] Error getting master registration for 127.0.98.61:40283: OK, instance_id { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" instance_seqno: 1790586217471970 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:03:37.694227 22640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:03:37.700783 22897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:37.701042 22897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:37.704449 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.694700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.0.98.62:37191)
0928 09:03:37.694759 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.694761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.695312 (+ 551us) server_negotiation.cc:407] Connection header received
0928 09:03:37.695408 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.695409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.695422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.695483 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.696241 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.697093 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.698106 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.698403 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.698471 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.700907 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.700909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.700909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.700916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.702772 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.703379 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.703380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.703381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.703439 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.703733 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.703734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.703734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.703888 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.703960 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.704273 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.704352 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:37.707509 22783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } attempt: 3
I20260928 09:03:37.709786 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.694966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.98.61:40283)
0928 09:03:37.695006 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.695008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.695467 (+ 459us) server_negotiation.cc:407] Connection header received
0928 09:03:37.696312 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.696313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.696323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.696353 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.698955 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.699685 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.702486 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.702810 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.702851 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.705416 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.705417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.705418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.705427 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.707361 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.708444 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.708445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.708445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.708502 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.708729 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.708730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.708730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.708872 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.709383 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.709583 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.709680 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 09:03:37.711776 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.695122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.0.98.60:32943)
0928 09:03:37.695158 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.695159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.697157 (+ 1998us) server_negotiation.cc:407] Connection header received
0928 09:03:37.698199 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.698200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.698210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.698238 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.700707 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.701414 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.704262 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.704564 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.704604 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.707997 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.707998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.707999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.708008 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.709895 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.710615 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.710616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.710616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.710693 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.710999 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.711000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.711000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.711146 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.711256 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.711594 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.711676 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:03:37.713660 22832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [term 1 FOLLOWER]: Refusing update from remote peer 060970a08cde40109618980383585edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:37.714028 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:37.715842 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 060970a08cde40109618980383585edd. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.715936 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 09:03:37 dist-test-slave-286n krb5kdc[22627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:37.720494 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.707774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32943 (local address 127.0.0.1:52928)
0928 09:03:37.707886 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:37.707897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:37.707933 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:37.710800 (+ 2867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:37.710801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:37.710948 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:37.710991 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.710993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.712359 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.712361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.713262 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:37.713264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.713855 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.713857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:37.713916 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.715653 (+ 1737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:37.715659 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:37.717218 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:37.719317 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.719318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.719319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.719580 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.719811 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:37.719812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:37.719812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:37.719872 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:37.720173 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:37.720174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:37.720219 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 09:03:37.720286 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:03:37.720389 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 09:03:37.720842 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:37.708002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.0.98.60:32943)
0928 09:03:37.710653 (+ 2651us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.710655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.710662 (+ 7us) server_negotiation.cc:407] Connection header received
0928 09:03:37.710682 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.710683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.710694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.711056 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.711098 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.712272 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.713358 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:37.713672 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:37.714069 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:37.717325 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:37.717327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:37.717327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:37.717335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:37.719211 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.719673 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.719674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.719674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.719736 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:37.719944 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:37.719945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:37.719945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:37.720081 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:37.720146 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:37.720622 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 09:03:37.720735 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":101,"threads_started":1}
I20260928 09:03:37.722527 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: Bootstrap starting.
I20260928 09:03:37.723366 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:37.723712 22783 log.cc:824] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:37.725226 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: No bootstrap required, opened a new log
I20260928 09:03:37.726181 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.726279 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:37.726349 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.726413 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:37.726675 22903 mvcc.cc:205] Tried to move back new op lower bound from 7334241147753058304 to 7334241147669364736. Current Snapshot: MvccSnapshot[applied={T|T < 7334241147753058304}]
I20260928 09:03:37.727432 22897 catalog_manager.cc:1347] Generated new cluster ID: 2b0c53ce11e44874899e24e89b32bac0
I20260928 09:03:37.727480 22897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:37.727450 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.727602 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4387312e3aeb4a9baf17860ecca645de, State: Initialized, Role: FOLLOWER
I20260928 09:03:37.727767 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } }
I20260928 09:03:37.728360 22756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:37.729341 22756 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [term 1 FOLLOWER]: Refusing update from remote peer 060970a08cde40109618980383585edd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:37.729643 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:37.728268 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:37.732708 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "060970a08cde40109618980383585edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4387312e3aeb4a9baf17860ecca645de"
W20260928 09:03:37.734297 22923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:37.734364 22923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:37.736719 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.736819 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:37.736873 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 060970a08cde40109618980383585edd. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.736923 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:37.736958 22909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.737003 22909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:37.737380 22915 mvcc.cc:205] Tried to move back new op lower bound from 7334241147753058304 to 7334241147669364736. Current Snapshot: MvccSnapshot[applied={T|T < 7334241147753058304}]
I20260928 09:03:37.737910 22757 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 060970a08cde40109618980383585edd for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 09:03:37.763885 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "060970a08cde40109618980383585edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060970a08cde40109618980383585edd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37191 } } peers { permanent_uuid: "4387312e3aeb4a9baf17860ecca645de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40283 } } peers { permanent_uuid: "9413fd1ae09848df992ffcf4e78d3cc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32943 } } }
I20260928 09:03:37.764047 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:37.766786 22897 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:37.768160 22897 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:37.776733 22897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 060970a08cde40109618980383585edd: Generated new TSK 0
I20260928 09:03:37.777011 22897 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:03:37.806809 22711 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.805043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.0.98.62:37191)
0928 09:03:37.805090 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.805092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.806076 (+ 984us) server_negotiation.cc:407] Connection header received
0928 09:03:37.806188 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.806189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.806201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.806250 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.806743 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55960: BlockingRecv error: recv got EOF from 127.0.0.1:55960 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 09:03:37.808033 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.805416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.0.98.61:40283)
0928 09:03:37.805460 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.805462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.807246 (+ 1784us) server_negotiation.cc:407] Connection header received
0928 09:03:37.807559 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.807560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.807571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.807601 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.807968 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48328: BlockingRecv error: recv got EOF from 127.0.0.1:48328 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 09:03:37.808768 22902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.805793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.0.98.60:32943)
0928 09:03:37.805852 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.805854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.807483 (+ 1629us) server_negotiation.cc:407] Connection header received
0928 09:03:37.807535 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.807536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.807554 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.807640 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.808699 (+ 1059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52930: BlockingRecv error: recv got EOF from 127.0.0.1:52930 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 09:03:37.811336 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.810582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.0.98.61:40283)
0928 09:03:37.810617 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.810618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.810822 (+ 204us) server_negotiation.cc:407] Connection header received
0928 09:03:37.810934 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.810936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.810954 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.810983 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.811275 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48338: BlockingRecv error: recv got EOF from 127.0.0.1:48338 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 09:03:37.814395 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.813619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.0.98.61:40283)
0928 09:03:37.813652 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.813654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.813907 (+ 253us) server_negotiation.cc:407] Connection header received
0928 09:03:37.814003 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.814004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.814014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.814043 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.814351 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48352: BlockingRecv error: recv got EOF from 127.0.0.1:48352 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 09:03:37.818210 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.817486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.98.61:40283)
0928 09:03:37.817526 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.817528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.817748 (+ 220us) server_negotiation.cc:407] Connection header received
0928 09:03:37.817849 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.817850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.817859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.817885 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.818166 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48358: BlockingRecv error: recv got EOF from 127.0.0.1:48358 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 09:03:37.824142 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.823346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.61:40283)
0928 09:03:37.823381 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.823383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.823635 (+ 252us) server_negotiation.cc:407] Connection header received
0928 09:03:37.823742 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.823743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.823754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.823782 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.824092 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48362: BlockingRecv error: recv got EOF from 127.0.0.1:48362 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 09:03:37.834316 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.833456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.61:40283)
0928 09:03:37.833498 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.833500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.833794 (+ 294us) server_negotiation.cc:407] Connection header received
0928 09:03:37.833889 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.833890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.833901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.833952 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.834258 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48366: BlockingRecv error: recv got EOF from 127.0.0.1:48366 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 09:03:37.852181 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.851463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.98.61:40283)
0928 09:03:37.851512 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.851514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.851638 (+ 124us) server_negotiation.cc:407] Connection header received
0928 09:03:37.851716 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.851717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.851729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.851775 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.852109 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48382: BlockingRecv error: recv got EOF from 127.0.0.1:48382 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 09:03:37.886162 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.885289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.0.98.61:40283)
0928 09:03:37.885333 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.885335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.885609 (+ 274us) server_negotiation.cc:407] Connection header received
0928 09:03:37.885705 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.885706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.885718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.885746 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.886102 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48398: BlockingRecv error: recv got EOF from 127.0.0.1:48398 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 09:03:37.952316 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:37.951628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.61:40283)
0928 09:03:37.951706 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:37.951708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:37.951844 (+ 136us) server_negotiation.cc:407] Connection header received
0928 09:03:37.951881 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:37.951882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:37.951893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:37.951957 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:37.952268 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48404: BlockingRecv error: recv got EOF from 127.0.0.1:48404 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 09:03:38.082118 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:38.081478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.61:40283)
0928 09:03:38.081530 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.081532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.081608 (+ 76us) server_negotiation.cc:407] Connection header received
0928 09:03:38.081653 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.081654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.081664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.081714 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:38.082016 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48410: BlockingRecv error: recv got EOF from 127.0.0.1:48410 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 09:03:38.339936 22787 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:38.339292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.98.61:40283)
0928 09:03:38.339376 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.339378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.339389 (+ 11us) server_negotiation.cc:407] Connection header received
0928 09:03:38.339442 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.339443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.339455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.339528 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:38.339861 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48420: BlockingRecv error: recv got EOF from 127.0.0.1:48420 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 09:03:38.656215 22875 catalog_manager.cc:1259] Loaded cluster ID: 2b0c53ce11e44874899e24e89b32bac0
I20260928 09:03:38.656294 22875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: loading cluster ID for follower catalog manager: success
I20260928 09:03:38.657138 22875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: acquiring CA information for follower catalog manager: success
I20260928 09:03:38.657426 22875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9413fd1ae09848df992ffcf4e78d3cc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:03:38.734853 22923 catalog_manager.cc:1259] Loaded cluster ID: 2b0c53ce11e44874899e24e89b32bac0
I20260928 09:03:38.734951 22923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: loading cluster ID for follower catalog manager: success
I20260928 09:03:38.735795 22923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: acquiring CA information for follower catalog manager: success
I20260928 09:03:38.736111 22923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4387312e3aeb4a9baf17860ecca645de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:03:38.856086 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:38.853144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.98.61:40283)
0928 09:03:38.853280 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.853282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.853341 (+ 59us) server_negotiation.cc:407] Connection header received
0928 09:03:38.853442 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.853443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.853523 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.853611 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:38.853756 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.854448 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.855237 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.855461 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.855546 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:38.855725 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:38.855854 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:38.855943 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:38.855966 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 09:03:38.855987 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260928 09:03:38.860206 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:38.857990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.0.98.60:32943)
0928 09:03:38.858090 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.858092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.858100 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:38.858155 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.858156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.858190 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.858256 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:38.858321 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.858780 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.859503 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.859701 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.859755 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:38.859874 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:38.859984 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:38.860053 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:38.860073 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 09:03:38.860090 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":57,"threads_started":1}
I20260928 09:03:38.865124 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:38.861948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.0.98.62:37191)
0928 09:03:38.862055 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.862058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.862182 (+ 124us) server_negotiation.cc:407] Connection header received
0928 09:03:38.862227 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.862228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.862276 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.862354 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:38.862487 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.863263 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.864228 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.864552 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.864609 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:38.864785 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:38.864893 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:38.864958 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:38.865009 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 09:03:38.865034 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260928 09:03:38.865499 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:38.862128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.98.61:40283)
0928 09:03:38.862171 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.862173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.862205 (+ 32us) server_negotiation.cc:407] Connection header received
0928 09:03:38.862316 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.862317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.862329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.862385 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:38.862576 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.863139 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.864699 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.864976 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.865034 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:38.865216 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:38.865254 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:38.865306 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:38.865398 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 09:03:38.865421 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:38.865577 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:38.862309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.98.60:32943)
0928 09:03:38.862348 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:38.862350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:38.862498 (+ 148us) server_negotiation.cc:407] Connection header received
0928 09:03:38.862596 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:38.862597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:38.862606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:38.862650 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:38.862768 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.863440 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.864194 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:38.864482 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:38.864545 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:38.864906 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:38.864960 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:38.865015 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:38.865485 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 09:03:38.865506 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:03:38.866832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22640
I20260928 09:03:38.872327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22715
I20260928 09:03:38.877302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 22790
2026-09-28T09:03:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2813 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 09:03:38 dist-test-slave-286n krb5kdc[23007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 09:03:38 dist-test-slave-286n krb5kdc[23007](info): set up 2 sockets
Sep 28 09:03:38 dist-test-slave-286n krb5kdc[23007](info): commencing operation
krb5kdc: starting...
W20260928 09:03:40.024963 392 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/test-user.keytab.
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T09:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T09:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 09:03:40.141536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33231,127.0.98.61:44553,127.0.98.60:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 09:03:40.231040 23020 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:03:40.231278 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:40.231307 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:40.231346 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:40.232450 23020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:40.232517 23020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:40.232542 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:40.232573 23020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:40.232602 23020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:40.234068 23020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33231,127.0.98.61:44553,127.0.98.60:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:40.234346 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:40.234591 23020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:40.239217 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:40.239226 23026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:40.239241 23025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:40.239493 23020 server_base.cc:1034] running on GCE node
I20260928 09:03:40.239717 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:40.239987 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:40.241146 23020 hybrid_clock.cc:648] HybridClock initialized: now 1790586220241130 us; error 26 us; skew 500 ppm
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:40.243501 23020 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 09:03:40.243948 23020 webserver.cc:460] Webserver started at http://127.0.98.62:42239/ using document root <none> and password file <none>
I20260928 09:03:40.244190 23020 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:40.244258 23020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:40.244408 23020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:40.245539 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data/instance:
uuid: "da9317f8e92a4b42bc9d853222621979"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "6cf0c9edaa22a1caf4337fac169cba4f"
server_key_iv: "8624c8c52bdedd59e22dc89ab6de69fb"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.245894 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal/instance:
uuid: "da9317f8e92a4b42bc9d853222621979"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "6cf0c9edaa22a1caf4337fac169cba4f"
server_key_iv: "8624c8c52bdedd59e22dc89ab6de69fb"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.247170 23020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 09:03:40.247869 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:40.248034 23020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:03:40.248131 23020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal
uuid: "da9317f8e92a4b42bc9d853222621979"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "6cf0c9edaa22a1caf4337fac169cba4f"
server_key_iv: "8624c8c52bdedd59e22dc89ab6de69fb"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.248214 23020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:40.275121 23020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:40.277222 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:40.277420 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:40.281874 23020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33231
I20260928 09:03:40.281912 23087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33231 every 8 connection(s)
I20260928 09:03:40.282302 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/data/info.pb
I20260928 09:03:40.283313 23088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:40.285447 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:40.286324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 23020
I20260928 09:03:40.286458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-0/wal/instance
I20260928 09:03:40.286662 392 external_mini_cluster.cc:1428] Setting key 46dae3c780088be0de1955863cb69065
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 09:03:40.296379 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.286269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.98.62:33231)
0928 09:03:40.286500 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.286510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.286624 (+ 114us) server_negotiation.cc:407] Connection header received
0928 09:03:40.287394 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.287396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.287422 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.287506 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.288302 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.289675 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.290545 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.290846 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.291026 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.293092 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.293098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.293099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.293115 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.294705 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.295159 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.295162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.295162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.295243 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.295481 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.295482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.295483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.295687 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.295780 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.295961 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.296141 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260928 09:03:40.296384 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.286200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33231 (local address 127.0.0.1:37386)
0928 09:03:40.286449 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.286501 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.287192 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.287542 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.287545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.287926 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.288167 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.288173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.289759 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.289760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.290441 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.290443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.291008 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.291009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.291073 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.291625 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.291633 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.292986 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.294804 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.294810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.294812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.295066 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.295319 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.295320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.295320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.295415 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.295758 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.295763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.295825 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.295938 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.296188 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":126,"thread_start_us":101,"threads_started":1}
I20260928 09:03:40.297269 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 09:03:40.297628 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.297523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60170)
0928 09:03:40.297556 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.297597 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 09:03:40.297770 23039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
I20260928 09:03:40.298153 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.298078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60176)
0928 09:03:40.298108 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.298132 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 09:03:40.298259 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44553: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 09:03:40.311738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44553
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33231,127.0.98.61:44553,127.0.98.60:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:03:40.347496 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attempt: 1
I20260928 09:03:40.348039 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.347875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60186)
0928 09:03:40.347969 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.348007 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 09:03:40.348456 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.348364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60198)
0928 09:03:40.348416 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.348441 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 09:03:40.348555 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44553: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
W20260928 09:03:40.392469 23095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:03:40.392741 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:40.392772 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:40.392815 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 09:03:40.393944 23095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:40.394014 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:40.394037 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:40.394058 23095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:40.394088 23095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:40.395620 23095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33231,127.0.98.61:44553,127.0.98.60:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:40.395911 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:40.396144 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:40.400967 23100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:40.401036 23103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:40.400930 23101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:40.401093 23095 server_base.cc:1034] running on GCE node
I20260928 09:03:40.401499 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:40.401813 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:40.403033 23095 hybrid_clock.cc:648] HybridClock initialized: now 1790586220403024 us; error 36 us; skew 500 ppm
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:40.405881 23095 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 09:03:40.406384 23095 webserver.cc:460] Webserver started at http://127.0.98.61:37243/ using document root <none> and password file <none>
I20260928 09:03:40.406618 23095 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:40.406687 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:40.406862 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:40.407931 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data/instance:
uuid: "5d91eee4fe1c4d3bad21991613e9f0bf"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "97b5a10be11d672bee5306656cb8dc44"
server_key_iv: "6bc75bdacd53294c84acb7fafeef09f4"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.408299 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal/instance:
uuid: "5d91eee4fe1c4d3bad21991613e9f0bf"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "97b5a10be11d672bee5306656cb8dc44"
server_key_iv: "6bc75bdacd53294c84acb7fafeef09f4"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.409711 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 09:03:40.410492 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:40.410667 23095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:03:40.410776 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal
uuid: "5d91eee4fe1c4d3bad21991613e9f0bf"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "97b5a10be11d672bee5306656cb8dc44"
server_key_iv: "6bc75bdacd53294c84acb7fafeef09f4"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.410852 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:40.416716 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attempt: 2
I20260928 09:03:40.417104 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.416977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60210)
0928 09:03:40.417037 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.417076 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 09:03:40.417452 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.417354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60226)
0928 09:03:40.417397 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.417422 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 09:03:40.417532 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44553: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44553: connect: Connection refused (error 111)
I20260928 09:03:40.420485 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:40.422854 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:40.423019 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:40.427762 23095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44553
I20260928 09:03:40.427790 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44553 every 8 connection(s)
I20260928 09:03:40.428221 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/data/info.pb
I20260928 09:03:40.429311 23163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:40.430784 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:40.436590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 23095
I20260928 09:03:40.436738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-1/wal/instance
I20260928 09:03:40.436950 392 external_mini_cluster.cc:1428] Setting key bd9f8b21cb374d01c4792c4f4692f66e
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:40.440930 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.431493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.0.98.62:33231)
0928 09:03:40.431566 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.431568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.431787 (+ 219us) server_negotiation.cc:407] Connection header received
0928 09:03:40.432245 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.432246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.432258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.432346 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.433183 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.433674 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.434673 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.434879 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.434941 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.437782 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.437784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.437784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.437793 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.439367 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.439857 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.439858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.439859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.439910 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.440187 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.440187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.440187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.440333 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.440384 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.440782 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.440863 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 09:03:40.441188 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.431474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33231 (local address 127.0.0.1:37388)
0928 09:03:40.431653 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.431695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.432159 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.432433 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.432436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.432869 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.433086 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.433090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.433760 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.433761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.434585 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.434587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.434975 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.434976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.435085 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.435583 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.435599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.437682 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.439446 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.439453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.439455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.439780 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.439977 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.439977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.439978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.440123 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.440403 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.440410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.440590 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.440738 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.440933 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260928 09:03:40.441820 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:40.451843 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.442103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60242)
0928 09:03:40.442195 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.442207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.442253 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.442464 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.442466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.442632 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.442704 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.442705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.443695 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.443696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.444581 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.444584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.445162 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.445163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.445303 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.445704 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.445709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.448460 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.450465 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.450465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.450466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.450700 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.451009 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.451010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.451010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.451089 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.451432 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.451433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.451485 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.451570 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.451693 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 09:03:40.451987 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.442168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.0.98.61:44553)
0928 09:03:40.442307 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.442311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.442330 (+ 19us) server_negotiation.cc:407] Connection header received
0928 09:03:40.442360 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.442362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.442377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.442440 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.442807 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.443600 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.444724 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.445050 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.445174 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.448573 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.448582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.448583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.448597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.450365 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.450871 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.450874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.450874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.450935 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.451165 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.451166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.451167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.451358 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.451428 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.451637 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.451825 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 09:03:40.452718 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:40.453106 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.452964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40874)
0928 09:03:40.453018 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.453074 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 09:03:40.453265 23113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
I20260928 09:03:40.453742 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.453631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40880)
0928 09:03:40.453682 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.453714 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 09:03:40.453869 23163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35425: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 09:03:40.463965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaJshIb/build/release/bin/kudu
/tmp/dist-test-taskaJshIb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35425
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33231,127.0.98.61:44553,127.0.98.60:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 09:03:40.503139 23163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } attempt: 1
I20260928 09:03:40.503657 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.503497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40894)
0928 09:03:40.503568 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.503618 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 09:03:40.504122 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.504012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40908)
0928 09:03:40.504067 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.504097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 09:03:40.504256 23163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35425: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
I20260928 09:03:40.508699 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } attempt: 3
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:40.516422 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.509112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.98.61:44553)
0928 09:03:40.509152 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.509154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.509169 (+ 15us) server_negotiation.cc:407] Connection header received
0928 09:03:40.509267 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.509268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.509290 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.509337 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.509711 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.510269 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.511274 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.511491 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.511542 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.513636 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.513638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.513638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.513646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.515267 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.515635 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.515636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.515637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.515698 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.515896 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.515897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.515898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.516014 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.516086 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.516246 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.516326 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:03:40.516443 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.509044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60246)
0928 09:03:40.509115 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.509128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.509198 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.509350 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.509351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.509554 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.509628 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.509631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.510364 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.510365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.511192 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.511194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.511562 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.511563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.511639 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.512118 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.512124 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.513547 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.515370 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.515371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.515371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.515549 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.515766 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.515767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.515767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.515830 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.516089 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.516090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.516138 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.516221 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.516326 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 09:03:40.517283 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:40.517678 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.517564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40916)
0928 09:03:40.517616 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.517643 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 09:03:40.518057 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.517947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40922)
0928 09:03:40.518003 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.518034 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 09:03:40.518144 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35425: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
W20260928 09:03:40.548841 23170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 09:03:40.549049 23170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 09:03:40.549067 23170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 09:03:40.549121 23170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 09:03:40.549285 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } attempt: 1
I20260928 09:03:40.549741 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.549604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40934)
0928 09:03:40.549669 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.549711 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 09:03:40.550154 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.550072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40944)
0928 09:03:40.550113 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.550138 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:03:40.550264 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35425: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
W20260928 09:03:40.550581 23170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 09:03:40.550623 23170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 09:03:40.550638 23170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 09:03:40.550652 23170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 09:03:40.550670 23170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 09:03:40.552300 23170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33231,127.0.98.61:44553,127.0.98.60:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 08:05:41 UTC on ccycloud.quasar-wvjmel.root.comops.site
build id 4676817
I20260928 09:03:40.552500 23170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 09:03:40.552742 23170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 09:03:40.557672 23178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:40.557775 23170 server_base.cc:1034] running on GCE node
W20260928 09:03:40.557760 23175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 09:03:40.557717 23176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 09:03:40.558099 23170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 09:03:40.558425 23170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 09:03:40.559581 23170 hybrid_clock.cc:648] HybridClock initialized: now 1790586220559555 us; error 43 us; skew 500 ppm
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 09:03:40.562189 23170 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 09:03:40.562635 23170 webserver.cc:460] Webserver started at http://127.0.98.60:36013/ using document root <none> and password file <none>
I20260928 09:03:40.562867 23170 fs_manager.cc:359] Metadata directory not provided
I20260928 09:03:40.562938 23170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 09:03:40.563097 23170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 09:03:40.564172 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data/instance:
uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "5939797fcd7b111fa3face68518d78b1"
server_key_iv: "5e468870c935a1d2963ffe326c3944cb"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.564567 23170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal/instance:
uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "5939797fcd7b111fa3face68518d78b1"
server_key_iv: "5e468870c935a1d2963ffe326c3944cb"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.565961 23170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 09:03:40.566641 23185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 09:03:40.566809 23170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 09:03:40.566915 23170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal
uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606"
format_stamp: "Formatted at 2026-09-28 09:03:40 on dist-test-slave-286n"
server_key: "5939797fcd7b111fa3face68518d78b1"
server_key_iv: "5e468870c935a1d2963ffe326c3944cb"
server_key_version: "encryptionkey@0"
I20260928 09:03:40.566987 23170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 09:03:40.572407 23163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } attempt: 2
I20260928 09:03:40.572839 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.572720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40952)
0928 09:03:40.572773 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.572810 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 09:03:40.573168 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.573076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40954)
0928 09:03:40.573120 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.573146 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 09:03:40.573249 23163 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35425: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35425: connect: Connection refused (error 111)
I20260928 09:03:40.583681 23170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 09:03:40.586315 23170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 09:03:40.586503 23170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 09:03:40.590528 23170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35425
I20260928 09:03:40.590556 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35425 every 8 connection(s)
I20260928 09:03:40.590912 23170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/data/info.pb
I20260928 09:03:40.591866 23238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 09:03:40.593427 23238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } has no permanent_uuid. Determining permanent_uuid...
I20260928 09:03:40.599191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaJshIb/build/release/bin/kudu as pid 23170
I20260928 09:03:40.599303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaJshIb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585920139418-392-0/minicluster-data/master-2/wal/instance
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 09:03:40.599478 392 external_mini_cluster.cc:1428] Setting key 73135355e7513b3589d0e4427ba7529b
I20260928 09:03:40.602916 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.594083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.0.98.62:33231)
0928 09:03:40.594170 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.594172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.594347 (+ 175us) server_negotiation.cc:407] Connection header received
0928 09:03:40.594832 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.594833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.594845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.594888 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.595489 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.595964 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.597073 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.597502 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.597634 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.600015 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.600017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.600017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.600025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.601541 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.601960 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.601961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.601962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.602024 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.602280 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.602282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.602282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.602402 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.602477 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.602745 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.602823 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 09:03:40.603117 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.594027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33231 (local address 127.0.0.1:37392)
0928 09:03:40.594212 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.594254 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.594726 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.594965 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.594974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.595211 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.595392 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.595396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.596076 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.596077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.596979 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.596982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.597657 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.597659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.597801 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.598302 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.598310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.599892 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.601644 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.601651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.601653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.601882 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.602091 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.602092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.602093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.602206 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.602469 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.602475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.602614 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.602724 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.602880 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260928 09:03:40.603778 23238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:40.608554 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.599879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.0.98.60:35425)
0928 09:03:40.600037 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.600042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.600081 (+ 39us) server_negotiation.cc:407] Connection header received
0928 09:03:40.600110 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.600112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.600126 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.600191 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.600845 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.601534 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.602297 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.602595 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.602675 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.605392 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.605399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.605400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.605406 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.607273 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.607616 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.607619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.607620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.607681 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.607892 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.607893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.607893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.608109 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.608247 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.608352 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.608433 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260928 09:03:40.610998 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.604052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44553 (local address 127.0.0.1:60248)
0928 09:03:40.604099 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.604109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.604145 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.604296 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.604297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.604432 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.604477 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.604478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.605211 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.605212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.605994 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.605997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.606471 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.606472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.606543 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.606840 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.606845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.608188 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.610044 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.610045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.610046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.610248 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.610440 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.610440 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.610440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.610489 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.610723 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.610724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.610772 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.610821 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.610912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 09:03:40.611021 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.604145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.98.61:44553)
0928 09:03:40.604199 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.604201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.604209 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:40.604232 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.604232 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.604243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.604295 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.604555 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.605120 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.606076 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.606390 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.606449 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.608279 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.608280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.608281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.608288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.609972 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.610313 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.610314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.610314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.610373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.610545 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.610546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.610546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.610665 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.610727 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.610879 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.610963 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 09:03:40.611363 23238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:40.617713 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.609504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.98.62:33231)
0928 09:03:40.609563 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.609565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.609573 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:40.609610 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.609611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.609619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.609688 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.610112 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.610800 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.612200 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.612488 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.612577 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.615139 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.615140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.615141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.615148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.616528 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.616933 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.616934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.616934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.616992 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.617189 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.617190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.617190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.617301 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.617496 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.617540 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.617612 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 09:03:40.618209 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.611619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40976)
0928 09:03:40.611656 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.611666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.611696 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.611801 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.611801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.611935 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.611965 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.611966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.612860 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.612861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.613525 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.613527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.614024 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.614025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.614074 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.614394 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.614399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.615707 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.617257 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.617258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.617258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.617419 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.617607 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.617608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.617608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.617659 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.617910 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.617910 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.617941 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.618022 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.618105 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 09:03:40.618319 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.611670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.0.98.60:35425)
0928 09:03:40.611700 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.611702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.611730 (+ 28us) server_negotiation.cc:407] Connection header received
0928 09:03:40.611747 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.611750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.611762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.611797 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.612050 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.612805 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.613604 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.613946 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.614015 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.615787 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.615788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.615788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.615794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.617168 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.617491 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.617492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.617492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.617543 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.617726 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.617726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.617727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.617836 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.617890 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.618083 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.618160 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 09:03:40.619665 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: Bootstrap starting.
I20260928 09:03:40.620481 23238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:40.620855 23238 log.cc:824] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:40.621742 23238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: No bootstrap required, opened a new log
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 09:03:40.623548 23238 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.623711 23238 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f5b7caf9bc4dcdaa37057ee1cd6606, State: Initialized, Role: FOLLOWER
I20260928 09:03:40.623792 23238 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.624100 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.624158 23238 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:40.624207 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.625406 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } attempt: 2
I20260928 09:03:40.626019 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.618543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.0.98.61:44553)
0928 09:03:40.618619 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.618621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.618629 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:40.618671 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.618672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.618681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.618734 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.619154 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.619632 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.620344 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.620550 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.620622 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.622997 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.622998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.622999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.623006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.624513 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.624974 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.624975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.624975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.625024 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.625264 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.625265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.625265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.625414 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.625574 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.625818 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.625905 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 09:03:40.626104 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:40.626155 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:40.628037 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 09:03:40.627028 23170 master_runner.cc:186] Error getting master registration for 127.0.98.62:33231: OK, instance_id { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" instance_seqno: 1790586220280572 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:03:40.629295 23170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:03:40.629099 23062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "da9317f8e92a4b42bc9d853222621979"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37396
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:40.633402 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.625757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:40984)
0928 09:03:40.625833 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.625845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.625897 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.626065 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.626066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.626225 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.626270 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.626272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.627218 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.627219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.627989 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.627993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.628547 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.628548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.628604 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.629046 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.629051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.630403 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.632201 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.632202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.632203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.632423 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.632682 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.632683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.632684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.632767 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.633097 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.633098 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.633144 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.633211 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.633304 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 09:03:40.633412 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.625775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.0.98.60:35425)
0928 09:03:40.625811 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.625812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.625883 (+ 71us) server_negotiation.cc:407] Connection header received
0928 09:03:40.625957 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.625958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.625967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.626019 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.626494 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.627142 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.628093 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.628413 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.628469 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.630502 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.630504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.630504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.630511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.632092 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.632498 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.632500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.632500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.632559 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.632866 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.632867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.632867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.633026 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.633100 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.633252 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.633326 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 09:03:40.635028 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979: Bootstrap starting.
I20260928 09:03:40.635721 23088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:40.636016 23088 log.cc:824] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:40.636834 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979: No bootstrap required, opened a new log
I20260928 09:03:40.638226 23088 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.638363 23088 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9317f8e92a4b42bc9d853222621979, State: Initialized, Role: FOLLOWER
I20260928 09:03:40.638437 23088 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.638690 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.638734 23088 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:40.638747 23062 raft_consensus.cc:482] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 09:03:40.638823 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.638872 23062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:40.639799 23062 raft_consensus.cc:504] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.640064 23062 leader_election.cc:288] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [CANDIDATE]: Term 1 election: Requested vote from peers 5d91eee4fe1c4d3bad21991613e9f0bf (127.0.98.61:44553), e4f5b7caf9bc4dcdaa37057ee1cd6606 (127.0.98.60:35425)
I20260928 09:03:40.640404 23212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da9317f8e92a4b42bc9d853222621979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606"
I20260928 09:03:40.640533 23212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [term 0 FOLLOWER]: Advancing to term 1
W20260928 09:03:40.640683 23269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 09:03:40.640726 23269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:40.641389 23212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da9317f8e92a4b42bc9d853222621979 in term 1.
I20260928 09:03:40.641943 23038 leader_election.cc:302] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da9317f8e92a4b42bc9d853222621979, e4f5b7caf9bc4dcdaa37057ee1cd6606; no voters:
I20260928 09:03:40.642066 23258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 09:03:40.642212 23258 raft_consensus.cc:686] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [term 1 LEADER]: Becoming Leader. State: Replica: da9317f8e92a4b42bc9d853222621979, State: Running, Role: LEADER
I20260928 09:03:40.642293 23258 consensus_queue.cc:237] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.642712 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9317f8e92a4b42bc9d853222621979. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.642904 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:40.642859 23020 master_runner.cc:186] Error getting master registration for 127.0.98.61:44553: OK, instance_id { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" instance_seqno: 1790586220426185 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 09:03:40.645126 23020 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 09:03:40.660017 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.642596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.98.60:35425)
0928 09:03:40.642648 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.642651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.645413 (+ 2762us) server_negotiation.cc:407] Connection header received
0928 09:03:40.645545 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.645546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.645570 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.645633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.646987 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.647728 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.649282 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.649583 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.649649 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.654836 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.654838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.654838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.654846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.657514 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.658501 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.658503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.658503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.658560 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.659343 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.659343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.659344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.659472 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.659577 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.659826 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.659901 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 09:03:40.660224 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.642755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.98.62:33231)
0928 09:03:40.642842 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.642844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.645318 (+ 2474us) server_negotiation.cc:407] Connection header received
0928 09:03:40.645572 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.645573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.645583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.645612 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.649315 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.650043 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.653217 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.653514 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.653555 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.656349 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.656350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.656351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.656358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.657889 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.659020 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.659022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.659022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.659077 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.659359 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.659360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.659360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.659480 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.659558 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.660055 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.660128 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 09:03:40.663647 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.642870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.0.98.61:44553)
0928 09:03:40.642910 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.642912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.647514 (+ 4602us) server_negotiation.cc:407] Connection header received
0928 09:03:40.647604 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.647605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.647616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.647645 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.650950 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.651681 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.655665 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.656001 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.656042 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.660748 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.660750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.660750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.660758 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.662436 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.662810 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.662811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.662812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.662875 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.663075 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.663076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.663076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.663201 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.663277 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.663475 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.663553 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 09:03:40.664335 23163 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } attempt: 3
I20260928 09:03:40.661712 23280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 09:03:40.664884 23280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 09:03:40.666673 23212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [term 1 FOLLOWER]: Refusing update from remote peer da9317f8e92a4b42bc9d853222621979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:40.667071 23275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 28 09:03:40 dist-test-slave-286n krb5kdc[23007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790586220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 09:03:40.668987 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9317f8e92a4b42bc9d853222621979. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.669073 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.672315 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.664593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35425 (local address 127.0.0.1:41012)
0928 09:03:40.664720 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 09:03:40.664731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 09:03:40.664773 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 09:03:40.664941 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 09:03:40.664942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 09:03:40.665094 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 09:03:40.665136 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.665138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.665999 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.666000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.666824 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 09:03:40.666835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.667301 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.667302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 09:03:40.667374 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.667760 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 09:03:40.667765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 09:03:40.669204 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 09:03:40.671219 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.671220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.671221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.671447 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.671669 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 09:03:40.671670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 09:03:40.671670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 09:03:40.671733 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 09:03:40.672015 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 09:03:40.672015 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 09:03:40.672073 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 09:03:40.672139 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 09:03:40.672232 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 09:03:40.672588 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:40.664680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.0.98.60:35425)
0928 09:03:40.664839 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.664841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.664847 (+ 6us) server_negotiation.cc:407] Connection header received
0928 09:03:40.664867 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.664868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.664877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.664929 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.665206 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.665891 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.666912 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:40.667183 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:40.667246 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:40.669419 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 09:03:40.669420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 09:03:40.669421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 09:03:40.669427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 09:03:40.671096 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.671529 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.671530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.671530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.671590 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 09:03:40.671798 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 09:03:40.671799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 09:03:40.671800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 09:03:40.671928 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 09:03:40.672000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:40.672178 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 09:03:40.672261 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260928 09:03:40.674137 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: Bootstrap starting.
I20260928 09:03:40.674880 23163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: Neither blocks nor log segments found. Creating new log.
I20260928 09:03:40.675190 23163 log.cc:824] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: Log is configured to *not* fsync() on all Append() calls
I20260928 09:03:40.676059 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: No bootstrap required, opened a new log
I20260928 09:03:40.677665 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.677769 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:40.678117 23280 catalog_manager.cc:1347] Generated new cluster ID: 990fbc52b8b744779fe375e96b9abbcb
I20260928 09:03:40.678181 23280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 09:03:40.678159 23163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.678267 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.678303 23163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d91eee4fe1c4d3bad21991613e9f0bf, State: Initialized, Role: FOLLOWER
I20260928 09:03:40.678363 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.678419 23163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } }
I20260928 09:03:40.678732 23163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 09:03:40.678788 23137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da9317f8e92a4b42bc9d853222621979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf"
I20260928 09:03:40.678892 23137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 09:03:40.679809 23137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da9317f8e92a4b42bc9d853222621979 in term 1.
I20260928 09:03:40.678709 23282 mvcc.cc:205] Tried to move back new op lower bound from 7334241159848341504 to 7334241159751888896. Current Snapshot: MvccSnapshot[applied={T|T < 7334241159848341504}]
I20260928 09:03:40.680127 23136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [term 1 FOLLOWER]: Refusing update from remote peer da9317f8e92a4b42bc9d853222621979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 09:03:40.681437 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.681530 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.681803 23275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 09:03:40.684731 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9317f8e92a4b42bc9d853222621979. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.684818 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.684921 23287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.684970 23287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 09:03:40.689421 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da9317f8e92a4b42bc9d853222621979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9317f8e92a4b42bc9d853222621979" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } peers { permanent_uuid: "5d91eee4fe1c4d3bad21991613e9f0bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44553 } } peers { permanent_uuid: "e4f5b7caf9bc4dcdaa37057ee1cd6606" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35425 } } }
I20260928 09:03:40.689524 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979 [sys.catalog]: This master's current role is: LEADER
I20260928 09:03:40.689806 23298 catalog_manager.cc:1259] Loaded cluster ID: 990fbc52b8b744779fe375e96b9abbcb
I20260928 09:03:40.689862 23298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: loading cluster ID for follower catalog manager: success
W20260928 09:03:40.689950 23298 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 09:03:40.689977 23298 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 09:03:40.690294 23280 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 09:03:40.691022 23280 catalog_manager.cc:1504] Loading token signing keys...
I20260928 09:03:40.697587 23280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da9317f8e92a4b42bc9d853222621979: Generated new TSK 0
I20260928 09:03:40.697788 23280 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 09:03:40.727723 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.726078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.0.98.61:44553)
0928 09:03:40.726124 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.726127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.727058 (+ 931us) server_negotiation.cc:407] Connection header received
0928 09:03:40.727164 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.727165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.727177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.727207 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.727651 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60258: BlockingRecv error: recv got EOF from 127.0.0.1:60258 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 09:03:40.728689 23091 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.726489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.98.62:33231)
0928 09:03:40.726529 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.726531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.728177 (+ 1646us) server_negotiation.cc:407] Connection header received
0928 09:03:40.728273 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.728274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.728285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.728314 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.728614 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37412: BlockingRecv error: recv got EOF from 127.0.0.1:37412 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:03:40.729511 23242 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.726769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.0.98.60:35425)
0928 09:03:40.726821 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.726823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.729004 (+ 2181us) server_negotiation.cc:407] Connection header received
0928 09:03:40.729107 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.729108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.729118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.729146 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.729454 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41016: BlockingRecv error: recv got EOF from 127.0.0.1:41016 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 09:03:40.731889 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.731145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.0.98.61:44553)
0928 09:03:40.731178 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.731180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.731352 (+ 172us) server_negotiation.cc:407] Connection header received
0928 09:03:40.731470 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.731471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.731492 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.731522 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.731844 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60268: BlockingRecv error: recv got EOF from 127.0.0.1:60268 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 09:03:40.734871 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.734134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.0.98.61:44553)
0928 09:03:40.734165 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.734167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.734397 (+ 230us) server_negotiation.cc:407] Connection header received
0928 09:03:40.734491 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.734492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.734501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.734529 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.734832 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60274: BlockingRecv error: recv got EOF from 127.0.0.1:60274 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 09:03:40.738577 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.737849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.0.98.61:44553)
0928 09:03:40.737879 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.737881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.738089 (+ 208us) server_negotiation.cc:407] Connection header received
0928 09:03:40.738174 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.738175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.738184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.738210 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.738542 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60290: BlockingRecv error: recv got EOF from 127.0.0.1:60290 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 09:03:40.744557 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.743699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.98.61:44553)
0928 09:03:40.743735 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.743736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.744020 (+ 284us) server_negotiation.cc:407] Connection header received
0928 09:03:40.744121 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.744122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.744133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.744163 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.744504 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60298: BlockingRecv error: recv got EOF from 127.0.0.1:60298 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 09:03:40.754776 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.753859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.0.98.61:44553)
0928 09:03:40.753901 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.753903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.754186 (+ 283us) server_negotiation.cc:407] Connection header received
0928 09:03:40.754296 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.754297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.754310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.754342 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.754715 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60310: BlockingRecv error: recv got EOF from 127.0.0.1:60310 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 09:03:40.773294 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.772243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.0.98.61:44553)
0928 09:03:40.772287 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.772290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.772579 (+ 289us) server_negotiation.cc:407] Connection header received
0928 09:03:40.772761 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.772763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.772775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.772818 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.773230 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60316: BlockingRecv error: recv got EOF from 127.0.0.1:60316 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:03:40.807448 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.806644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.0.98.61:44553)
0928 09:03:40.806689 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.806691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.806851 (+ 160us) server_negotiation.cc:407] Connection header received
0928 09:03:40.806954 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.806955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.806968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.807037 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.807390 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60318: BlockingRecv error: recv got EOF from 127.0.0.1:60318 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 09:03:40.873317 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:40.872468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.0.98.61:44553)
0928 09:03:40.872536 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:40.872539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:40.872676 (+ 137us) server_negotiation.cc:407] Connection header received
0928 09:03:40.872777 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:40.872778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:40.872789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:40.872841 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:40.873248 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60332: BlockingRecv error: recv got EOF from 127.0.0.1:60332 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 09:03:41.003240 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:41.002532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.0.98.61:44553)
0928 09:03:41.002596 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.002598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.002737 (+ 139us) server_negotiation.cc:407] Connection header received
0928 09:03:41.002783 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.002784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.002795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.002847 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:41.003180 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60342: BlockingRecv error: recv got EOF from 127.0.0.1:60342 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 09:03:41.261323 23167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 09:03:41.260542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.0.98.61:44553)
0928 09:03:41.260592 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.260594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.260661 (+ 67us) server_negotiation.cc:407] Connection header received
0928 09:03:41.260764 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.260765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.260780 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.260863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 09:03:41.261252 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60344: BlockingRecv error: recv got EOF from 127.0.0.1:60344 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 09:03:41.626576 23255 catalog_manager.cc:1259] Loaded cluster ID: 990fbc52b8b744779fe375e96b9abbcb
I20260928 09:03:41.626675 23255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: loading cluster ID for follower catalog manager: success
I20260928 09:03:41.627559 23255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: acquiring CA information for follower catalog manager: success
I20260928 09:03:41.627874 23255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4f5b7caf9bc4dcdaa37057ee1cd6606: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:03:41.691283 23298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: acquiring CA information for follower catalog manager: success
I20260928 09:03:41.691596 23298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d91eee4fe1c4d3bad21991613e9f0bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 09:03:41.776885 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:41.774425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.0.98.61:44553)
0928 09:03:41.774571 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.774573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.774586 (+ 13us) server_negotiation.cc:407] Connection header received
0928 09:03:41.774613 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.774613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.774663 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.774733 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:41.774868 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.775358 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.776094 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.776279 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.776329 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:41.776522 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:41.776625 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:41.776739 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:41.776802 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 09:03:41.776820 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 09:03:41.781106 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:41.778616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.0.98.60:35425)
0928 09:03:41.778739 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.778741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.778749 (+ 8us) server_negotiation.cc:407] Connection header received
0928 09:03:41.778862 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.778862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.778905 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.778973 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:41.779070 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.779696 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.780385 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.780614 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.780720 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:41.780830 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:41.780925 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:41.780977 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:41.781027 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 09:03:41.781043 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260928 09:03:41.785598 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:41.782621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.98.61:44553)
0928 09:03:41.782732 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.782735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.782950 (+ 215us) server_negotiation.cc:407] Connection header received
0928 09:03:41.783025 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.783026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.783036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.783090 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:41.783260 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.784007 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.784849 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.785136 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.785200 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:41.785340 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:41.785374 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:41.785425 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:41.785481 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 09:03:41.785500 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 09:03:41.785748 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:41.782841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.98.60:35425)
0928 09:03:41.782902 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.782904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.783134 (+ 230us) server_negotiation.cc:407] Connection header received
0928 09:03:41.783155 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.783156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.783165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.783211 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:41.783355 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.783810 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.784911 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.785201 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.785256 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:41.785401 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:41.785437 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:41.785481 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:41.785627 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 09:03:41.785651 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 09:03:41.785909 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 09:03:41.782631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.98.62:33231)
0928 09:03:41.782742 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 09:03:41.782744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 09:03:41.782866 (+ 122us) server_negotiation.cc:407] Connection header received
0928 09:03:41.782945 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 09:03:41.782946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 09:03:41.783005 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 09:03:41.783095 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 09:03:41.783188 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.783948 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.784961 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 09:03:41.785208 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 09:03:41.785269 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 09:03:41.785481 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 09:03:41.785594 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 09:03:41.785634 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 09:03:41.785803 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 09:03:41.785828 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260928 09:03:41.787472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 23020
W20260928 09:03:41.792346 23187 connection.cc:535] server connection from 127.0.0.1:40984 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 09:03:41.792889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 23095
I20260928 09:03:41.798013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaJshIb/build/release/bin/kudu with pid 23170
2026-09-28T09:03:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2920 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5734 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301664 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS